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detelopaentt and innovation (BD S Z) analysis fraaavork developed by 
Jrhe aathors. The studies are intended to deaonatrate the reqairesents 
and poasibilities for policy analyais in the BO S I arena and the 
utility of the fraaevozk for facilitatiaq such endeavors and for 
providinq atracture to the knoaledqe and literatora in this field. 
The first policy analysis* done for the Bational Institute of 
Bduoation CBIBU looks at reaearch* davelopaent* disseaination* and 
evaluation reaearch in the field of education and auqgests 
isplicatlons for the role of BIB. The second policy analyaia provides 
a aystess analysis of educational BO 5 Z evaluatinq the current atate 
Of the systea* The third policy analyais discusses and pointa out 
defieienciies in two reports. The f^cst report ia one done by the 
Bational Acadeay of sciences CBASI concerninq fundaaental research 
_»^y&At to education and written to halp the BZ2 fulfill ita 
reaponaibility in this area. The aecond report analyzed is a response 
to the BAS report done by the Proqraa Ccaaittee of the Bational 
Council on Educational Beaearch. Thia report plao contains a series 
of policy recoaaendations for BIE. Copies of both reports are 
included in the appendix. CAuthor/JB) 
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This volume addresses Itself to a difficult and complex set of Issues 
related to the making and implementing df policies with respect to ' 
Research* Development and Innovation (R/O&I) processes and systems. 
Despite the e^tence of large (if fragmented) literatures on policy 
and R/D&I relevant subjects, the complexity of R/O&I and i^licy making 
issues and processes continues to confound policy makers. Nev6r...§tasy» 
policy making in the R/D&I context can be especially perplexing for 
the policy maker* the max; jer and the analyst. Mot the least among 
the factors causing this condition is the uncertainty associated with 
such processes. This uncertainty Is manifested in goal setting 
(we generally lack precise targets; goals are frequently emergent); 
In resource requirements (it is usually very difficult to estimate 
the how much, what kind and how long parameters of what it will take 
to pursue a project or program); and consequently » in evaluation 
aborts and cost/benefit considerations (it is often difficult to 
obtain agreement as to what benefits are relevant; to measure and 
evaluate the impact of programs or projects; or to assign and measure 
costs) . 'because we are deiTllng in futures (often quite long-term) , 
the process is subject to numerous and potentially dramatic consequences 
of changes in the environment (social, political, technological, 
ecological, etc.> in which the work goes on; thus, forecasting 
becomes an latportant yet difficult aspect of policy. Because we are 
dealing with innovation. Issues of response to change and the complexity 
of knowledge processes come into play. 

The meaning of the concept of R/D&I is discussed later in this intro- 
duction. 



T%% sources and locations of the Innovation process (and Its components) 
are of major policy concern* ■ Is the process to be driven by technolo- 
glcal opportunity or by user need (often described In push/pull terms)? 
Vho Is /should be involved in the process? Vhat Is the nature of the 
actual and/or needed Interaction between producers and users? What 
is the impact of such interaction (or the lack of interaction)? 
Requirements for dealing withfj^dissemlnation, implementation and dif- 
fusion of innovations thus become a part of policy concerns. How 
do these producer/tiser and push/pull Issues vary as a consequence of 
the nattire of the field, the nature of the knowledge base and the 
maturity of the R/D&I system and its institution^. The various com- 
ponents of an R/O&I process (the research, the development, the dis- 
semination, etc*) tend to have substantially different characterlstldk 
and policy and management requlrementip; thus, there arise Issues of 
linkage and balance between and across these Ri[D&I process components. 
The above are not well tind4rstood phenomena, and they make policy 
making difficult. 

Stich issxies arise at the institutional level (generally thought of in 
terms of R&D management in business and governmental organizations), 
and at system sectoral, natlQnal end even international levels. Thus, 
inter^organlzational issues arise such as those Involved in the R/D&I 
systems in education, in law enforcement, in an industry, in energy, 
^<d.th respect to the role of technology development ^or the third world, 
etc. In such cases, the con^plexity of the policy issues is Increased 
immeasurably. 

Finally, we may note that we are dealing with policy questions that 
will Involve proitessionals and professional organizations. Policies 
and management procedures that ignore the implications of -this -fact 
in terms of goal setting, decision orocesaes, motivational patterns, 
information flows, reference groups etc., are doomed to fall. 

In this volume we will deal with several) such policy Issues > using a 
contextual analysis framework thatCe have developed « This framework 
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riftflecu a synthesis of the research and policy literatures on 
R&D oanagement, the economics of R&D» Innovation, the sociology 
of science, the Qax^gement and policy sciences, and of many, 
years of ipanagerlal and research experience with such questions. 
A brief description of this framework is given below. For a 
more complete description the reader should turn to a companion 
volume* (Research. Development and Innovation t Contextual 
Analysis 1977). ' 

The policy is^s we have dealt with and which are presented and 
analysed in sohaaquent chapters of this book were for the most 
part presented to us as majyr R/D&I concerns for the organisa- 
tions involved* We had litple or nothing to say in defining the 
initial problem statement » although we had |mich to say in terms 
of red^ifining the problems for analysis into forms that had use- 
ful and policy actionable isq^llcatloiiis. We then went on to use 
our analytical framework to develop policy and ma n a ge ments 
requirements in each case. It is our. hope that in presenting 
these, we can demonstrate both the requirements and pdsslbilitles 
for policy analysis in the KjtDStl arena, and the utility Of our 
framework for facilitating' stich endeavors and for providing some 
structure to our scattered knowledge (and literatures) in this 
field. Further » we believe that this framework makes an Impor- 
tant contrlbu^tion to dealing with the problems of comparative 
policy analysis across fields and sectors by showing h'^. one can 
and must deal with contextual differences, and therefore by im- 
plication; with the consequences for attempts to transfer ex- 
perience and methods from one sector to another (e.g.: as be- 



Three volumes have been prepared to illustrate the nature and 
utilization of our contextual analysis framework. Each Is di- 
rected to one of three specific audiences: (1) policy analysts 
and policy makers; (2) researchers; and (3) the educational 
R/D&I coosQunity* 



tween industry and education and aerospace, etc.). Compara- 
tive analysis of policy issues also illustrates differences be- 
tween r/d&I in sociul science contexts and R/D&I in physical 
or life science contexts as well as between specific sectors. 
These differences can also be critical in attenpts to transfer 
clxperience and methods across sectors. These last points are 
elaborated in some detail in the previously cited comparison 
volume. 

\ 

Before goin^ further, let us consider some of the more general 
' issues of policy making and then return to the specific ar^na for 
this book, policy making in Pesearch, Development and Innovation, 



\ 
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% I . POLICY MmNGt DIFFICULT YEI IMPORTANT 

« 

' ^Ti|l»iaaklng of poXlcitfS Is at best (as we have noted) a difficult 

business. In any ^Iven Instance, a policy maker Is faced with 

. s 

. such critical issues as: ' - ^ 

- the sufficiency, adequacy, avall<:kbillty of needed Infoxnatlon; 

- choices among alternative policies, ^ach of which will likely 
r \ ' have different, (and perhaps conflicting) ramifications; Is 

likely to have critical limitations; will likely Involve a 
set of assumptions, about conditions which must and/or do exist, 
V . the expected impact of the policy^ the ♦'values'Vgoals/purposes 

t^lch the policy ia to serve, etc.; 



1* 

the resources required to lii9)lement a n^^llcy^ 

a variety of uncertainties such as tb jicertalntles of 
economic, political and soclo-cultural realities and dynamics; 
the uncertainties asspciated with diffuse and loosely linked 
systems (as in the social science context) ; the lack of clarity 
and consensus over g6als at the level of nadonal systems and 
of large complex organizations; etc.; 



identifying, comprehending and "sorting out" the broad, inter- 
active complex of factors and dynamics which may be relevant for 
any given policy Issue; 

the potential Interactive effects among a variety of policies 
within a single Institution or between a set of Institutions 
which may be loosely linked (if at all) and whose Interests and 
purposes (and hence policies) ,may at times be either conflicting 
or complementary; 
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- constraints which niuat» In effect » be accepted as "glvens". 

\ 
I 

1 

Policy making, then, is not an easy task. The complexities and 
uncertainties tend to be of large magnitudes. Relevant conditions 
change ov6r time — sometimes in a very brief period of time. 
Variables are often non*quantifiable (except perhaps in a very in- 
direct, "indicator" sense). The policy maker, as a human being, 
brings to the policy making task particular assumptions, biases, 
views of "reality*!, values, goals, etc. (and those may be per- 
sonal, organizational and/or cultural in nature) . The policy 
maker will generally be provided (or confronted) with "partisan" 
information, advice, perspectives (Cain 1971) — and will sim- 
ilarly be faced with various "power" realities (e.g. : political 
power that the Congress has over the funding — and even the 
existence — of a federal agency). 

While policy making is not an easy task, it is nonetheless an 
important task; Cain describes policies as "discrete, parti- 
cular, authorative, content-laden decisions which indeed come out 
of some process and presumably have critical ramifications for 
somebody, and often everybody." While arguing^ that the complexity 
may be such that analysis can never be "good enough" to "solve" 
a policy problem, Lindblom (1968) also argues that policy ana- 
lysis does have an impact — and more so the better it is done. 
And we may simply note that the absence of a policy also has 
ramifications and Impact. 
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IX« POLICY SCIENCEt INSIGHTS IKTO THE PROCESS OF POLICY ANALYSIS 

« 

In light of the Import^mce yet difficulty of the policy making 
process, it is little wonder that over the past decade or so, a 
i field of policy science has been developing — in effect, as an 
"effort to cope with the needs of a complex, dynamic and demanding 
society'* (George Washington University, 1968-1969) . This field 
has not yet matured and thus does not yet have a well-defined 
identity or inAge (Lasswell 1970) — indeed, Radnor, White axyl 
Tansik have offered several "definitions" of policy science 
(or aspects of policy science): a process for handling com- 
plex policy issues by including consideration of "political, 
' institutional, •irrational' (but real), and other nonquanti- 
fUble and value- laden factors"; a descriptive study; improve- 
ment in the design of the policy-making process; a set of 
skills, people, etc.; and just "something responsive to ill- 
structured complexity" (Radnor, White and Tansik 1975). We may 
draw some significant insights into the nature of policy ana- 
lysis from this developing field of policy science. 

1. Complex issues must be examined in the light of their total, 
complex, interactive contexts. Complex issues sJlmply cannot ^ 
be "effectively dealt with in a partial, specialized and arbitrary 
mannef (George Washington University 1960-1969) . The implication 
here is two-fold: 

a) 



b) 



Policy analyses must consider a broad range of dynamics, 
factors, contextual conditions, etc. 

The context for a policy issue must be analysed from multiple 
perspectives (e.g.: the perspectives of various disciplines 
and professions; the perspectives of those who make, those 
who must implement and those who would be impacted by a . 
policy; the perspectives of varying, multiple purposes 
which might be relevant to a policy; etc.) 
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In a word, the context used as a basis for analysis must be 
"sufficiently comprdxenslve to identify significant relation* 
ships" (George Washington University 1968-1969). 

2. At the sane time, policy analysis must be "sufficiently limited 
to be managed effectively" (George Washington University 1968- 
1969), and to be useable for the policy maker — I.e., policy 
analysis must be focused. 

3. The function of policy analysis Is to provide a "handle" 
that enables a policy maker to "take hold" pf a policy 
Issue — l.e.» policy analysis "must provide decision- 
makers with a basis for more confident control over 
events by assisting In the articulation of critical con- 
tingencies," (George'^shlngton University 1968-1969) 
conditions, dynamics, factors, opportunities, constraints, 
etc* 

4. Policy analyses may be seen as a process for Identify- 
ing and analysing alternatives (George Washington Uni- 
versity 1968-1969): alternative descriptions of the 
context of a. policy Issue; alternative goals and ob- 
jectives; alternative ways of conceptualizing an Issue; 
alternative courses of actions; alternative predictions 
of probable outcome^ of certain courses of action are 
taken; alternative methods for measuring, obtaining 
data about and evaluating policy outcomes; etc. 

♦ 

5. Care must be taken to recognize the limitations of j 
policy analysis In general and of specific policy ^alyses. 
For example: the policy analyst will tend to bri^g' 
certain personal, organizational and/or socio-cultural 
assumptions and perspectives to the task of analysis. 

The "givens" of a society may significantly pre-deter- 
mine the shape of a policy (George Washington University 




1968*1969) . Policy analysis nay be validly seen as an 
attempt to bring "rationality** to bear on a policy issue — 
yet "rationality" is limited by the complexity involved,-^* 
by the non-quantifiable and value-laden nature o£ policy 
issues, by lack of information, by political considera- 
tions,^ a lack of well-defined goals, and even by the 
very meaning of rationality" « (Dror, 1975). Lindblom 
argues, in effect, that policyjiaking is and will be 
an imperfect process: "man's design of the (policy 
making) system never controls specifically or precisely — 
but only within broad limits . . ." 



The impact of litical considerations may call for a "different" 
kind of rationality than we normally mean ^Aien we speak of "being 
rationale". 

Dror comments that the term "rationality" is used either "as a 
synonym for 'reasonable' > or as referring to estimation ability, 
or as referring to operations -research- like capability or — 
usually -- as referring to a mixture of all these possible meanings." 

Lindblom notes a distinction between "gooij analysis" and "part- 
isan nature of analysis" views of the nature and process of policy 
anal^^sis. 
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III. THE CHALLENCS OF R/P&I POLICY ANALYSTS 

Thus £ar» we have talked about policy analysis and the policy process 
from a general level perspective - - I.e., at the "generic" level. 
Our concern in this volume, however. Is as we stated earlier, somewhat 
more specific. We are here concerned with policy analysis relevant to 
Research, Development and Innov^ion (R/D&I). 



-^e 
nM 



e use of the term "R/O&I" is deliberate. One of the major weak- 
ses In R/D&I policy making has been that "R«?*' — the production 
of new knowledge (KP) and knowledge products — is given an essentially 
Elusive and myopic emphasis. Inadequate consideration tends to 
be^vftiven (by those concerned with R&D systems and policy Issues) to 
the ^act that R&D is only a part of a more total process of innovation - 
a process which includes not only the knowledge production functions 
of research and developmeiit but "downstream" knowledge utiligation (KU) 
functions of acquisition, implementation, utilisation as well as 
KP-KU linking functions such as need identification, dissemination/ 
diffusion/marketing/distribution, and evaluation research. Thus, to 
connote this total process of innovation of which R&D is a part,, we 
use the term R/D&I/— Research, Development and Innovation. 



Another major problem that has tended to plague R/D&I policy analysis 
and policy making has been an inability to deal with' contextual 
coiqplexlty on the one hand^and, relatedly, the ccmplex interplay of 
R/D&I issues on the other hand. Thus, we tend to find that policy 
analysis flounders at either of two extremes. At one extreme, R/D&I 
policy issues have been approached from the particularistic perspec- 
tives of specific theories, specific disciplines, specific institu- 
tional perspectives — perspectives which are too specific to comprehend 
the complexity involved in a policy isstie and which thus can neither 
identify the range of relevant variables and alternatives nor "sort 
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out** the most critical variables and alternatives. At the other 
extreme, policy Issues have been approached from a "total world" 
perspective which attempts to cover everything and leave out nothing 
with ^ predictable result that policy analysis gets bogged down 
In a complex morass (of Issues, factors, dynamics, details, alternatives, 

♦ 

contlnge^eles, etc. The policy maker is then left with a certainty 
that there exists a very big problem — but with little else. The 
policy maker faasyno "handles", no indication of which factors are most 
critical, which factors can be impacted, or what particxilar policies 

and strategies arls relevant or ^feasible. 

i 

Similar comments can be made in relation to R/D&I program and project 
selection processes. ^1)ownstream" utilization Issues tend to be\^ 
ignored. Significant variances are often ignored because assumptixms 
are made that et^aln factors will remain constant, when in fact \ ^ 
such assumptions may be quite in error. 

Thus, the need in R/D&I policy analysis is for some process or 
mechanism which allows the policy analyst to consider the total 
process of innovation; to Identify and take into account the inter- 
plays among policies and policies issues, betwecm institutions, etc*; 
to consider a broad range of contextual conditions and variables 
(e.g., institutional, personnel and knowledge/ technology bases; 
political, legal, economic, social environments; information flows; 
historical development Issues; etc.); — and at the same time to 
focus analysis on those considerations which are most critical for 
the policy issue at hand. 



\ 
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IV. A FRAMEWORK FOR CONTEXTUAL ANALYSIS OF R/D&I 

I" 

Wltihout trying to debate whether any policy analysis can ever be 
''completely rational" or "coBq>letely comprehensive" (though we have 
our doiibts)» we do believe that the policy analyst must have some 
analytical framework which allows the analyst to take Into consider* 
at Ion the full, broad ranxe of contextual complexity and then to 
identify those factors whldti are most critical In order to limit 
and focus the analysis. Such a framework must be able to take into 
considexiation, separately and in interaction, those factors and 
dynamics which are generic to R/D&I and those which are (descriptively) 
specific or peculiar to the particular R/O&I system and/ or issue at 
hand. Such a framework must be able to utilize varying perspectives 
and to be utilized by personnel having differing perspectives. 

Over the past two years, the Northwestern University Center for the 
Interdisciplinary Study of Science and Tecinology (CISST) has been 
attempting to develop such a framework—*- i^ich we call a "contextual 
analysis framework" — afid it is the purpose of this report both to 
describe this framework and to illustrate its usage in policy-relevant 
ways. • 

. Before turning to a discussion of this contextual analysis framework, 
it is critical that it! be recognized for what it is — and what it is 
not. A contextual analysis framework is a tool for analysis — speci- 
fically, it provides a means of access into analysis which allows the 
analysis to be cooqprehensive, yet focused. It is not a substitute for 
analysis, or for knowledge about the subject matter at hand; nor does 
it guarantee the validity, cff fcacy or relevancy of a policy analysis. 



♦A more complete discussion of this framework is found in, Radnor, 
Michael, Harriet Spivak and Durward Hofler, "Research, Development 
and Innovation: Contextual Analysis", December 1917. 



The nature and adequacy of policy analysis reaalns the domain of the 
policy analyst — - Just as pollqr decisions remain the domain of the 

it 

policy maker. 

1. The Perspective of Oeneral Systems Theory 

The framework presented here for the annlysls of R/D&I systems 
has been dravn from the general systems theory literature. With- 
^ out attempting to present an exposition of this perspective, we 
simply note that we have adopted the central elements of their ^ 
fraaawork for describing the structure and functioning of living 
systems.' 

Thus, we will attempt to analyse R/O&l systems In terms of how 
they Interact with their environments; their ceiitral elements 
or sub-systems; the mechanisms that link them together; Internal 
system structures; Input-output systems as well as such other' 
system conditions as age apd state of developmental l^turlty. 

. • 

An Important question Is that of system definition: What is to be con- 
sidered within the R/D&I system (and within which par^of the system) 
and what is in the environment? The framework we arelpresentlng does 
not contAln abrupt boundary notions. What is considered within or 
external to the system and whether an R/D&£ system is defined broadly 
(as a whole sector) or narrowly (as a single Institution) is a matter 
of degree and will depend upon the focus and purpose of analysis. Thus 
rather fhan being limited by a rigid conceptualization about boundaries 
and sizes, the definition of the relevant R/D&I systems (as is true 
of all other aspects of the analytical framework) is bajjosd on c^ mating 
an opportunity to fvme key questions related to the focus of the 
issue analyses relevant to policy/decision makers and researchers. 



The discussion which follows, is derived, with slight modification 
from Radnor, Spivak and Hofler, 1977. 
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IWo aspects of a systems perspective on R/D&I are worth noting here. 



A. Extent of R/p&I System Linkage 



One aspect of the "system" concept that should he considered here 
Is the extent to which the various Institutions within an R/D&I 
system are (or are not) coherently and strongly linked together. 
Simply put* i4ille we do consider a set of institutions to comprise 
an R/D&I system (hecause of the roles they play within the total 
process of innovation) , we do not presume tnat they are in fact 
coherently linked together in "appropriate" ways or that existing 
linkages are strong. Indeed » the opposite may be t^e in any 
given context — and there may be "gaps" in the system's linkages. 
Indeed* the critical issues here are precisely the nature* Ftrength 
and appropriateness of the linkages (or lack thereof) which do 
exist. . 

B. Maturation (State of Development) 

It is important that we understand R/D&I systems from ain "organic" 
perspective. That is to sa/, that they "emerge" over time, that 
they go through/may be at different stages or levels .of development 
(maturation). Further, different it*stit itiom* and/or different 
functions within an R/D&I systom may differ in terms of tljieir 
respective stages or levels of development. The importance oi 
this concept of maturation may be aeen in at least the following 
ways: 



1. The nerds of an R/D&I system may be different when the 
system is young and itamature than when it is established 
and mature* 

2. Since R/D&I systems may mature (pr decline) over time, 

I 

their needs may change over time. 
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3. Policies* strategies and mechanisms whl^h are relevant 
for an R/D&I system which Is young and Immature may 
be^'lrrelevant (even dysfunctional) for an R/D&I system 
which ^ established and mature. 

4. Further, different pclicles/^trategies/mechanlsms may 
be needed when the R/D&I institutions and/or functions 
are at different stages of development than whyi their 
levels of development are "In balance". 

Ve need* however* to understand that the concrete meaning of 
"maturation" nay differ significantly across sectors. Thus, for 
exaaqtle, even at a *Wture". stage of development we would not 
expect to.^ind the same level of clarity and certainty in the 
evaluation researc^ function In a social science sector such as 
education as we would in a p* /slcal science sector. The reali' 
satlon of inherent differences between sectors will be ioq>ortant 
if we are to avoid maklftg incorrect con^arisons of (and developing 
the wrong expectations for) one R/D&I system in relation to 
other R/D&I systems. 

♦ 

C. R/D&I System Variations and Commonalities 

\ 

In attenpting to develop ^ contextual analysis framework for 
R/D&I systems, one is immediately struck by the immense amount 
of observable variety in real-^orld R/D&I systems. Existing 
R/D&I systems vary in such matters as: clusterings of functions 
within a single organization (or even within a single organiza- 
tional unit"; existence and strength of institutional networks; 
overall lei $1 of system maturity; susceptibility to political 
influence; types of dissemination mechanisms and strategies; 
use and effectiveness of various management technologies; etc. 

R/D&I systems vary across sectors. Thus, for example, the R/n&I 
system in aerospace differs significantly from the R/D&I system 

♦ 
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in education. But the Issues are coupllcated even further because 
R/D^ systems also vary within a singly sector. Thus* iov example, 
within the private industrial world we encounter rather Afferent 
K/D&I systems for the oil and T.V. industries. In health; the 
R/D&I system for drugj and surgical procedures will vary. \ 

. • \ • 

In eplte of the iffloense amount of observable variety among ej^stlng 
R/D&I systems, these systems nonetheless do also seem to exhibit 
common characteristics characteristics that imply the existezipe 
of generic features of R/D&I systena. 

\ 

From a broad, overview perspective, we may note that all R/D&I 
systems involve some rather basic, common functions , such as 
research, development, production, dissemination, utilization, 
etc. The specific form, manner or configuration, of these functions 
may vary across sectors, but the functions themselves appear to 
be inherent in the overall innovation process of an R/D&I system. 

We may further note that for any single R/D&I function, there 
appear to be characteristics which are common (i.e., g^eric) 
to that function both across and within sectors. For example, 
the basic research function involves a high level of uncertainty 
ar.I unpredictability; involves extending the limits of the 
existing state of the art (a criteria for "excellence*' in basic '. 
research); often involves a long time-line (10, 20, even 50 
years). These characteristics, if Indeed generic, will have 
strong implications for policy making and management in R/D&I 

systems. 

In contrast, the function of (j^velopment Involves a significantly 
lower degree of uncertainty and unpredictability; Is concerned 
with utility and "product specifications" Instead of "ultimate" 
qualities: requires less highly specialized and more inter- 



ERIC 



disciplinary personnel; and generally tends to have a short 
to aoderate time-line (usually at least 3 to 5 years, sometimes 
longer). These^ characteristics » if indeed generic, will have 
strong Implications for policy making and management in R/O&I 
systems — but the implications will be significantly different 
from the implications relevant to basic research. Further, 
these differences between the basic research and development 
functions (and other functions as well) will have strong policy 
making/management implications concerning the integration/ 
cooitdination/orchestration of the various R/D&I system functions. 

♦ 

It is also if^rtant to note that a "real-world" R/D&I system 
emerges as jui interactive *Hiorking out** of generic R/D&I character- 
istics within a specific sectoral context — whether by deliberate 
design or not. 

Thus, both the researcher and the decision maker mast understand 
that R/D&I system features, issues and management policies/ 
strategies will have both g^eric and sectoral dimensions, and 
that these will all be in interaction with each other. 



For the decision maker, the Importance of understanding this 
*'emergent" nature of R/D&I systems is threefold. 

1. An underatandlng of the generic characteristics of 
R/D&I systems enables the decision maker to **zero in" 
on the areas of the sectoral context where the critical 
issues are likely to be and where in-depth analysis 
of the sectoral context Ic needed » 

2e An xmderstanding of the nature and uniqueness of oneS 
sectoral context provides a basis for learning from 
R/D&I systems In other sectors and for determining 
the adaptablllty/transferablllty of knowledge, methods, 




I 

^ • . ■ 

techniques, innovations, etc., from R/D&I systems in 
other sectors. 

3. From an understanding of the Interaction between 
generic and sectoral characWlstics, the decision 
/ maker has a basis for develo^ng policies and strategies 
which are bot!i generically functional and sector- 
specific. 

.2. A Framework for "M td-Level AnaTY«4a" ' 

Our contextual analjrsls framework serves to focus attention on a some- 
what neglected area of research and analysis, sometimes referred to 
aa »'«id^level" or "mid-range". As used here, this •«mid-level" refers 
to reaearch and analysis which is somewhere between the bro^ui level 
of general theory and t^ * narrow level of specific cases. 

At the general theory level, the purpose of research and analysis is 
to develop concepts and relationships which serve to describe .all 
situations (i.e., theories). The approach at this level is to develop 
processes of research and analysis whidi will uncover the broadly 
generalisable concepts and relationships. While Important, general 
level theory lacks the specificity which is needed by policy and 
decision makers. 

At the specific case level, the purpose, of research and analysis is 
to discover and demonstrate the uniqueness of each situation, and 
the approach to research and analysis is designed specifically to 
uncover such uniqueneSl^ At this level, research and analysis tends 
to lack bases for generallzability. Thus, this level of research 
and analysis also has limited value for policy and^decislori makers. 

Our contextual analysis framework utilizes a disciplined conf Igurative 
a22roach that will permit systematic comparison of various sectors 
or policy/strategy issues. The purpose is to develop appropriate areas 
of generallMblllty which allow one to take into account the uniqueness 
of specific situations. 
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3. A flo ntexttial Framework: Describing R/D&I Systems In Terms of 
|helr Interactive Features 

In developing a fraaework for contextual analysis, we have Identified 
nineteen key R/D&I system features which we believe will be helpful 
both to the researcher and to policy analysts and to policy makers, 
(see Figure 1). For simplicity of presentation, we have grouped 
thesa nioeteen features Into the following categorical framework 
(using a general systems theory approach) : 

1. The R/D&I System's Environment 

f 

This category will include those features which are external 
to the R/DM system itself, but which may nonetheless 
impinge upon and affect the system — and, alternatively, 
which the R/D&I system may affect (e.g.: social, legal, 
political, economic, technological environments). 

2. Operative System Condltibns 

This category will include features internal to the R/D&I 
system jAitdt affect the way the system operates but which 
are not activities by which the system creates or utilizes 
knowledge. These features will thus include general system 
conditions (e.g.: historical development), aspects of 
system management (e.g.: administrative processes) and 
system Inputs and outputs (e.g.: personnel base). 

3. R/D&I Functions 

This category will Include those features which we would 
consider to be an Integral part of a knowledge production 
to knowledge utilization process continuum — i.e., what 
the system does to create and utilize knowledge. , 



\ 
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Typical Comparative features 



I» . ENVIROMMEST 



1. Environments of the R/D&X System 



II » OPSRATIVE SYSTEM 
CONDITIONS 



SYSTEM 



2. Historical Development 

3. Institutional Base (Network 
of Institutions) 



SYSTEM MANAGEMENT 



SYSTEM INPUTS AND 
OUTPUTS 



4* Goals, Policies, Strategies 

5. Administrative Processes 

6* PeM^nnel Base 

7. Funding 

8* InfArmatlon Flow 

9. Innovations 



HI. R/p&I FUNCTIONS 



IV. R/D&I RESEARCH 




10. Need Identification 

11. Genoratlon/Research 

12. Development 

13. Production 

14. Harketing/Dlstrlbutlon/Dls- 
semination/Diffusion 

15. Acquisition 

16. Implementation and Utilization 

17. Support Services 

18. Evaluation Research 

19. Research on R/D&I 



Figure 1 
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Comparative r/D&I System Features 

'28 
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Additloaally, we have Included an overview feature: Research on R/D&X. 
This will include any kind o£ research done about any aspect of the 
system (any of the features or feature Issues; any element of the 
system such as a particular Institution or set of institutions; etc.). 
The results of such research* in effect, provide the data base for 
analysis of the other features. 

Within each feature, a nuaiber of relevant issues nay be identified. 
An expanded discussion of these nineteen key R/D&I features and 
illustrative issues associated with each feature is provided in 
Radnor, Spivak and BOfler (1977) • 

A different^'^listlns or arrangenent of features and issiies could » of 
course* be developed. What is important Is to recognise, identify 
and analyze the potential or actual effects these various features 
nay have (separately and/or in interaction) in the total R/D&I system. 

4. The Context of an R/P&I System 

Taken together, the totality of the R/D&I system features and Issues 
forms an interactive context in which analysis and decision making 
must be performed. The way an R/D&I system has developed over time 
in its sectoral environment; the types of institutions that have 
4timerged; the character of the work and technologies; the personnel 
involved in each of the functions and institutions; etc. — - all 
contribute interactively to the totality of an R/D&I system's context. 

« 

For example > how an R/O&I system is structured will be Influenced by 
such factors as the social t political and economic environments of the 
^ institutions that constitute the R/O&I system; by the degree of 
system institutionalization; by the nature of the work to be performed; 
by the history and state of the system's development; by the nature 
of the system's personnel base; etc« But In turn> these same 
variables will also be influenced by tlie structure of the R/O&I 
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system. Such Is the Interactive nature of the R/D&I system context •*» 
each H/D&I system feature acts both as an Independent variable (as 
part of the total system context, affecting the other parts of the 
system) and as a dependent variable (which may be a focal concern 
for analysis and decision m^jklng). 

In any given Instance, analysis or decision making vlll, of course, 
be focused on some subset of contextual featuxes or Issues (or even 
on a single feature or a single Issue of a feature). Such a narrowing 
of focus la necessary to ^rlng the analytical/decision processes down 
to manageable and meaningful levels. Indeed, It Is la^ortant to 
recognise that each feature has Important characteristics which do 
distinguish one feature from another — differential characteristics 
i^lch often have liiq;>ortant iiiq>llcatlons for both analysis and decision 
making. 

\ 

However, the consideration of any single feature (or issue) must take 
into account the interaction of that specific feattore with all other 
features — i.e., one must consider a single feature or issue within 
the richness of its total context. To try to analyze any single 
feature (or issue) without considering Its contextual Interaction 
would not only be inadequate — it would likely be quite dysfunctional, 
leading to wrong conclusions by the analyst and to wrong decisions 
by the decision maker. Such is the interdependence within an inter- 
active living syston. 

Therefore a context has to be understood as the intersection of the 
effects or Influences of each of the system features. If we wish 
to tmderstan^ the character and managerial requirements of a given 
feature or issue (e.g.: the personnel base) it will be necessary 
to view this feature against the background of all other elements 
or features of the system in Its context. By the same token, if we 
ere concerned with a sub-issue within the personnel base feature (e.g. : 
the fl^w rate of certain types of personnel In and out of the system), 
then we would also have to include all the other aspects of the 
personnel base feature (e.g. : the types and levels of professionalism ) 
as part of the relevant context for that sub-lssue. 

30 
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V. EIGirr CdSTECTUAL R/D&I POLICY ANALYSES :\ A "COLLECTIVE " OVERVIEW 

In t;« chapters which follow are presenter eight policy analyses, 
each of which waa prepared separately and stands alone in its own 
right. Before looking at them individually, however, it is im- 
portant to understand what they represent collectively — the 
history and context out of which they come; how doing them and 
interacting with our "clients" has affected our own thinking; what 
they represent as policy analyses; our use of certain terms, 

1. History and Conteyt 

The contextual analysis framework discussed above and used in these 
policy analyses has hit^n "in the priocess" of development over the 
past 2% yeara though its roots may be traced back to our work 
.over many years ir the area we now call R/D&I. When the initial 
"rough draft" version of ^ the framework was reviewed and^iscussed 
in a workshop of HIE, field and CISST personnel (Spring, 1976)* 
the response was essentially: "It sot^nds interesting, but would 
it really be usable and useful for policy analysis?" We agreed 
that this was a valid question — • one. which could only be answered 
by actually demonstrating the use of the framework in specific 
policy analyses. We further agreed that for the demonstration to 
be valid, control over the selection of policy issues to be 
analyzed should not remain with CISST (like anyone else, we might 
tend to pick our "favorite" issues). Rather, the client agency 
should select those policy issuer which would be most relevant to 
itself. Finally, we agreed that it would be intport^nt to demon- 
strate the applicability of the framework across a variety of sectors** 
— as is represented in Chapters Six, Seven and Eight. 
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*The first presentation of the contextual analysis frantework in 
published form is found in Radnor, Spivak, Young and Hofler (1977), 
which was an abridged version of the more recevit volume on this 
subject (Radnor, Spivak and Kofler 1977). 

**In this regard, a companion volume (Radnor, Spivak and Hofler, 
1977) demonstrates the use of the framework to provide descriptive 
and comparative overviews of the contexts of four sectors: education, 
o civilian aviation, health and criminal Justice. 
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This oollactlon of policy analyses » then, represents an illustra- 
tion of the usability and usefulness of the contextual analysis 
framework in R/d&I policy analysis. 

^ As the reader will note, each of the policy analyses uses the 
contextual analysis franwwork in somewhat different ways — at 
least in earns of how its use has been "written up»*^in a particular » 
analysis. Thus, only the coordination analysis (Chapter Six) 
specifically refers to all nineteen R/D&l features. By contrast, 
the analysis of fundamental research (Chapter Three) does not 
directly mention the contextual analysis framework even though it 
guides the arguments presented. Other analyses focus on a selected 
set of the nineteen R/D&X features (e.g.: agency/field relations: 
Chapter One; assessment of educational R/D&I: Chapter Two; 
regionalism: Chapter Pour) — while the ERDA analysis (Chapter Seven) 
combines the nineteen R/D&l features into a more contact set of 
R/D&X features which make the proposed process more usable in the 
context of building an operational decision makeup procedure. 

These different ways of using the contextual analysis framework 
illustrate a critical point. The framework is a tool of analysis 
— not a straight jacket. It can — and indeed must — be used in 

4 

the way that is most fruitful for a specific policy issue or 
policy analyst. At the same time, w^ would be remiss if we did not 
emphasise that the perspective provl<ied b^^he:.^mew6 has permeated, 
directed and undergirded our work in ttkch analysis •>•> regardless 
of hdw "visible" its use may appear in each analysis. We have 
indeed found this perspective to lead to insights which would likely 
have otherwise been missed •>•> by us and by the client agency. 

2. A Growing Understanding^ of Contexts and Issues 

The eight policy analyses presented here were developed at various 
times over the past eighteen months. During this time, we have found 
that our understanding of particular contexts and Issues has both 
broadened and deepened — - and, Iti some Instances, thlD had led us 



to modify some of our earlier thinking. In part, this has come 
•through Interactive discussion of an analysis with a client agency 
and with others in the field a process which we deliberately 
build into the development of a policy analysis. In part, this 
has come through interaction among the different policy analyses 
— analyses which though separate and distinct nonetheless inform 
each other. 

To illustrate, in the agency/field analysis (October, 1976) we 
suggested a need for a central core of educational basic researchers, 
later, in the fundamental research analysis (August, 1977), we 
nodified our discussion of this idea to recognize that some 
dif fuseness in the field and lack of cumulativeness of research is 
natural in much of the socir.1 sciences. 

Similarly, we have modified our understanding of what a "mature** 
R/O&I field would "look like" in the social sciences. Our dis- 
cussidn of a mature R/O&I system in the agency/field analysis* leaves 
the impression that all mature R/D&I systems would have clear-cut 
and finely- tuned lines of specialization, linkages, communication 
channels and networks; that it would be "easy" to distinguish between 
|iigh quality and low quality R&D outcomes. Such a descriptlrA 
of maturity may be relatively accurate for R/D&I systems in the 
physical sciences (though even here we woul«^ hesitate to be "too 
absolutely clear-cut"). However, we now recognize that R/D&I 
systems in the social sciences — even in "mature" stages — will 
probably always be more diffuse and it will be less easy to develop 
a clear descriptive "map" of linkages or clean-cut "standards" for 
"high" and "low" quality R&D products. This latter point would be 
especially true in the educational R/D&I context, where (often 
conflicting) value and political Judgments cannot easily be 
"separated out" from an understanding of the meaning of "quality." 

r 

^Fages 209-210 of Chapter One in this volume. 



•In one sense, it would be tempt ln# to have revised aspects of the 
eight policy analyses (to reflect our more recent understandings) 
before presenting them in this Volume. To do 80» however, would 
be to gloss over and deny important aspects of the policy analysis 
process — namely, that policy analyses (as we noted earlier) is 
not and probably cannot be a **coii^lete and precise science"; 
and that it is both proper and necessary that any policy analysis 
be considered subject to '*iii|>rovement** through an interactive 
process of review and discussion. We hav» therefore deliberately 
chosen to present the eight policy analyses in the form in which 
they were initially presented to their respective client agencies.* 

3. Some Further Observations about Contextual R/D&I Policy Analysis 

This volume is about contextual R/D&I policy analysis. The chapters 
which follow demonstrate the use of contextual analysis in the 
R/D&I policy arena. This introduction has atteiq^ted to "set the 
stage** for reading the subsequent analyses by looking at the nature 
of policy analysis, of contextual analysis and of R/D&I. 

A few final observations about contextual R/D&I policy analysis 
must now be made. 

Each of the analyses which follow Include some set of conclusions 
which we believe to be important for policy/decision makers. These 
conclusions represent a blend of deductive reasoning Interacted 
with our own considerable experience in the R/D&I arena. This is 
precisely the way the contextual analysis framework Is designed — 
to allw an interactive process of deductive and inductive reasoning. 
We also recognise that alternative conclusions may sometimes be 
possible. This, too, is as it must be. Policy analysis simply is 

♦Chapter Six (on social science R&D coordination) and Chapter Eight 
(on development and the role of technology in developing countries) 
represent partial exceptions — i.^., the form of these analyses as 
presented here 1$ somewhat different from the original analyses, in 
both cases, however, the analyses represent only a change of format - 
and not of content — from the original analyses. 
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not A "precise science." Further yet, we do not consider either 
any Individual analysis In this volume or the volume as a whole to 
be the "fltial word" or a "finished product." More examples and 
scenarios are needed. Related Issues need to be ejq^lored. 
"Hissing" data needs to be collected. Conclusions need to be 
reviewed and tested. These areas represent future challenges — 
both to us and to the reader. 

It wilt also be Important for the reader to recognise that the role 
of policy analysis Is to help the policy/decision maker Identify 
critical Issues, ask the right questions, discover "blind spots," 
and avoid *Mumb pitfalls." It Is at this point that a contextual 
analysis framework comes Into play — by forcing consideration of 
. . the total R/D&I context while at the same tine permitting one to 
identify and focus attention on the most Critical areas for analysis 
and decision. 

It Is equally important to recognlae that while a policy analysis 
. may provide k basis for decision, it cannot take the final leap to 
'decision, i|n|^lementatlon, practice. This remains the domain of the 
polipy/decXsiOii maimer. 

Finally, it wlU^ be helpful to call attention to the meanings we 
ascribe to several terms which the reader will find throughout the 
analysis: 

When we speak of an R/D&I system , we are (as we noted earlier) 
' es^haslzlng the need to consider all parts of a total process 
of Innovation — i.e., to have a systems perspective. When 
we speak of an R/D&I "system," we do not mean to imply that 
it actually does or should exist in any "full blown," clearly- 
linked, centralized, authoritarian sense. In other words, 
the term "system" is used descriptively, not prescriptlvely *••> 
though of course analysis may lead one to conclusions about 
what kind of R/D&I system is feasible, desirable, etc. 
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When R/D&I is viewed from a broad-scdped perspective as a 
"system** or as a total process of Innovation, consideration 
must be given to svoh Issues as the need for balance among 
R/D&I functions, developing synergy among projects of various 
funding agencies and the like. We thus use the term 
orchestration both to connote such a broad-scoped perspective 
of system/Innovation process Interactions (and their Impacts) 
and to suggest that to suggest that such Interactions (and 
their impacts) need not necessarily be left to "r«ndom chance/* 
At the same time, the mode of and R/D&I system actors per- 
forming orchestration can vary (including nat urally emerging 
processes). Thus, while the concept of orchestration does 
imply purposeful, proactive actixdty from an overview per- 
spective, it is not meant to imply tight, centralized control. 

We recognize the distinction between research and technoloi^v 
on the one hand and science on the other — and that 
technology is not necessarily driven/ by science. 

R/D61I sji^tems do go through various stages of maturation . 
We do not posit a neat, clean-cut linear process of maturation 
— Indeed, different elements of an R/D&I system may be at 
different levels of maturation at any given point in time. 
At the same time, the characteristics, needs, requirements and 
capabilities of a mature R/D&I system are different than for 
an Immature R/D&I system — with potentially critical policy 
implications. 

In relation to R/D&I, the term users commonly refers to persons 
or Institutions in an operational, practice setting — i.e., 
those who use the outputs of knowledge production processes. 
For descriptive, categorizing purposes this broad distinction 
between knowledge producers and knowledge users Is valid. At 
the same time, it is Important to recognize that knowledge 
users can also be knowledge producers (and vice versa), 
perhaps especially in a sector such as education. 
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Another tani which appears in many o£^ the analyses is lead 
agency . Here, ve would make three comnents. First , the 
concept of a lead agency is one that "grew out of" our initial 
contextual R/D&t policy analyses (and in particular, from the 
agency/field analysis) •>«* we did not "bring" this concept to 
these analyses. Second , by "lead*' agency, we mean an agency 
whieh: (1) has an overW.ew perspective; (2) has (or is 
perceived to have) some kind of responsibility for an R/D&l 
system as a syst^; (3) has at least som^> resources with 
which to fulfill this responsibility; and (4) is one of 
several agencies or Institutions coneamisd with some aspect 
of R/06iI. Ve do not mean that: (1) any kind of monolithic, 
centralised control exists, would be possible, or is desirable; 
(2) ' any single agency would be the sole lead agency in a given 
sector (indeed, it is more likely that several agencies or 
institutions could be seen as lead agencies) ; (3) the agency ^ 
must be the major funding agency (NIE, for exaiqple, is not 
and yet it can be considered a lead agency. Third , we have 
in more recent analyses (e.g.: the coordination analysis in 
Chapter Six), taken more note of lead roles, in effect 
broadening the lead agency concept. 

The reader may note that all of these terms reflect a broad-scoped 
perspective of R/D&X. l^is is deliberate, it is a perspective 
which is inherent in the concepts of contextual analysis and of 
R/D&I.as a total process of innovation. It is also a perspective 
which, we believe, must inform nK)re narrowly-focused policy issues. 
At the same time, we recognize (and emphasize) that policy, strategy 
and management decisions and actions must also be informed by an 
understanding of more situation-specific specific dynamics, 
characteristics, needs, requirements, etc. Indeed, we etqphasize 
that contextual analysis is a process of Interacting broad-scoped 
and sitimtion-specific perspectives. The ERDA analysis (Chapter Seven) 
is an exati^le of such an interaction for ^the purpose of developing 
program planning (project selection procel^ses for use by a specific 
division of EROA). 



VI. EIGHT CONTEXTUAL R/D&I POLICY ANALYSES; BRIEF PREVIEWS 



Eavlng Ipokad at tha contextual R/D&I policy analyses 

from a collective perspective,. we will conclude this introduction 
by providing the reader with a brief suomary review of each 
of the eight analyses. 

1. Agency/Field Relationships in the Educational R/D&I System 
October 1976 

^8 presented initially to us by NIE, th^ concerns of the Agency 
focused on two questions of procurement policy: (1) the appro- 
priate balance between "field-initiated" vs. "NIE-directed" R&D; 
and (2) the appropriate mechanisms for procuring either field- 
initiated or NIE -directed work. Since these two questions are 
special cases of the broader and more critical issue of how NIE 
and the educational R/D&I field should relate, it was agreed 
that the policy analysis would focus on the agency/ fie Id issue* 

In thinking about this issue, we were struck by the rather fun- 
damental and broad -ranging implications of the questions raised, 
especially when viewed from the perspective of our undetstanding 
of R/D&I systems and processes and of the total, ^interactive pro- 
cesses in which R/D&I systems and processes exist and operate. 
Thus, there were some fundamental concerns which needed to be 
examined if the questions posed by NIE were to be responded to 
in an operational, policy and strategy relevant manner. These 
fundamental concerns included: 

1. the nature of NIE*s purposes and roles as a mission- 
oriented, lead agency in relation to educational R/D&I; 

2. the impact of NIE*s funding policies on NIE*s purposes 
(as these impact 9n the total educational R/D&I sector); 



3s 



the mxatlpllcity of NIB purposes (including, in addition 
to substantive R/D&l outputs: building educational R/D&I 
system capacity; affecting the system's environment; 
providing system stability; s>stem orchestration); 

a large aij/dlffuse operational system for education (l.e 
the users of educational R&D products) ; 

the relatively Inmature and loosely-linked nature of 
the educational R/D&l "system**; 

the differences in appropriate agency/ fie Id relations 
across the different R/D&I functions. 

We were also concerned with considerations of program and project 
**portf olios" that would permit synergy and orchestration both 
within NIB and within the educational R/D&I system; with latent 
as well as the manifest purposes implications of specific 
programs* projects, policies and strategies; with non-procurement 
as well as procurement policies and strategies; with the nature 
and implications of a variety of strategies by which an agency 
can relate to various parts of the field. 

In this analysis, we analyzed the agency/field issue separately 
for four key R/O&I functions: research (both basic and applied), 
development, dissemination and evaluation research* Each of these 
in turn, was analyzed in terms of (1) the generic nature of the 
function; (2) the educational context; and (3) the implications 
for the role of NIE. A comparative analysis was then performed 
across these four R/D&I functions. This comparative analysis re- 
vealed several common themes (most specifically: a requirement 
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ioT NIE leadership;' system building; and orchestration as the 
major MIS role) and some slgn:.f leant differences .(most specifically 
in r relation to time frames* the meaning of "excellence" for each 
function and key criteria for project selection). Finally, two 
brief scenario analyses were provided to illustrate how this policy 
analysis might affect policy and strategy decisions. 
\ 

\ 

2« Assessment of Educational R/p&I 
December 1976 

Institutionalised Research* Development and Innovation (R/D&I) ^ 
in education is little more than a decade old — yet the R/DCif 
system capacity (as we can assess it now)» our understanding 
of the system and our ability to manage it have increased slg'^ 

nificantly. \ 

\ 

There is now a need to develop and refine* over the next few 
years* an analytical framework and a relatively unobtrusive 
monitoring system (for data gathering) with which the educ 
tional R/D&I system could be assessed in terms both of pro- 
gress made to date and of what might reasonably be expected in 
the near term and longer term future. Such an assessment would 
provide the basis for annual or periodic reviews of the edu- 
cational R/D&I system. 

The analytical framework and the monitoring system for such 
assessment could be developed from a growing knowledge of 
R/D&I in other sectors and of the conditions pertinent to the 
education sector In particular. 

In this brief overview report, we have suggested in broad terms 
what such a framework might look like; what should be the basis 
for assessment in the current and succeeding ^periods ; what Is 
the current status of key elements in the system and reasonable 
near and longer term expectations for (based on the incomplete 
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and tentative evidence and impressionisttc judgements available ^ 
at the time); and finally what major needs require consideration 
in formulating federal polic:^ and program initiatives. 

This report is based on some key premises: 

1, that however weakiy linked or integrated, the insti- 
tutions and personnel involved in the production and 
^- utilisation of educational R/D&I outputs do form a 

"system" and not just a group of disaggregated entities; 

2» that R/D&I systems characteristically go through various 
stages of growth and devel6|iment , with different needs and 
dynamics being present at different stages of development; 

3. that over the past two decades, federal funding policies 

\ ■ \ 

have reflected )an increasingly broadening perspective o£ 

what constitutes an educational R/D&I system; 

V 

4. that these premises or perspectives have significant impli- 
cations for long term planning and monitoring and for the 
development of initiatives by a federal agency. 

This report has three parts: (1) an ai^sessment of the development 
of educational R/D&I system capabilities over the past two decades; 
(2) an assessment (including a discussion of the basis for assessment) 
of the current status and needs of major R/D&I functions (specifically: 
basic research, problem*-focused research, development, dissemination 
and evaluation research); and (3) a summary which suggests a general 
format for federal funding of educational R/D&I. 

While we do throughout the report note weaknesses in the educational 
R/D&I system, we also emphasize that the current state of the educa- 
tional R/D&I system must be assessed in terms of where it has been 
and where it now has the potential to go — not in terms of unrealistic 

4 ' 
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expectations about "progress and output to date"* Thus, it is 
itaportant to note that what we have found would be generally about 
what one would expect to have found within a reL^tively young R/D&I 
system. There are weaknesses, but there has been progress and there 
are signs of the beginnings of a transition from the introductory 
stages of development. 

3. Strengthening Fundamental Research Relevant to Education 
August 1977 • \ 

One part of NXE's overall re8pon8ibilit> is fundamental (basic) re- 
search relevant to education. In order to better fulfill this par^ 
of its responsibility, MIE sovght the advice and counsel of the 
National Acadeo^ of Sciences (NAS). Their advice and counsel were 
presented in a report ertitled: "Fundamental Research and the 
Process of Education" (1977). In response to the NAS report, the 
Program Cconittee of the National Council on Educational Research 
(NCER) presented a series of policy recommendations. Most notably, 
tUe NCER Program Committee recommended that 20% of NIE's total budget 
te allocated to fundamental research by 1979 and at least 30% by 1985.^ 

While agreeing with the NAS conclusions that fundamental re- 
search relevant to education does need to be strengthened, there 
were a number of significant and potentially very dysfunctional 
deficiencies in the NAS reporr, and ^sgecially in the NCER Pro- 
gr-un Committee recommendations. For example, inadequate con- 
sideration was given to: 

1) NIE's broad scope of responsibilities which. cover many 
areas relevant to e( ation — e.g.: development, 
dissemination, as well as the improvement of education 
as a practice — NIE cannot consider policies and 
strategies for fundamental research apart from Its 
other responsibilities. 
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2) the Impact that HIE policies and strategies regarding 
fundamental research could have on these other areas 
of MI£*s responsibilities; 

3) the policy and strategy implications of a number of 
critical aspects of the educational R/p&I context — 
e.g.: the ioqtortance to education o£ experience-based 
knowledge vis-a-vis research-based knowledge; the nature 
of the educational fundamental research personnel and 
institutional bases (number i quality; interest and com- 
mitment t9 education^ current capabilities to produc- 
tively use what levels of increased fundi^* how fast 
they can be "built up**); the relative inmaturity of 
educational R/S&I; 



4) the role of MIE as a goverzmiental, funding and **lead'* 
J agency in relation to the educational R/D&I context; 

5) rationales or criteria to guide the policy and stra- 
tegy deliberations of NCER and NIE. 

Most specifically, the NCER Program Committee does not provide 
a rationale for recommending that 20-30% of NIE's total budget 
be allocated to fundamental research. Indeed, this recommenda- 
tion appears to us to be highly dysfunctional when one consi- 
ders that NIE has many other major responsibilities; that the 
costs for such R/D&I functions are significantly higher for 
applied research and for development than for fundamental re- 
search; that such a level of resource allocation could not 
help but restrict NIE*s program planning flexibility. 

Finally, the HAS report could be interpreted as providing a 
rational (I) that NIE Is not needed as a funding agency; and/ 
or (2) that there Is no need to fund fundamental research 
whose focus is education per se. 
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This policy analysis, then, was developed to call attention to Issues 
such as those above; to.4iI^ce consideration of fund^inental research 
in a broader perspective o£ a total process of innovation, of NIE's 
more broadly scoped roles » and of the educational R/D&I context; 
and in so doing, to help provide a sounder basis both for the MAS 
coniBittee*s basic conclusion (that fundamental research relevant to 
education should be strengthened) and for policy/strategy decisions 
related to the conclusion. In this analysis, ve do suggest a fund- 
ing strategy as an alternative to the recommendations of the MCER 
Program Comoittee, as well as specific funding purposes and other 
non- funding strategies. 

4. Regionalism in the Educational R/D&I Context 
December 1977 

As with the issue of agency/field relations, the issue of regionalism 
in the educational R/O&I context was selected by NIE. The importance 
of regionalism as an issue for NIE can only be understood in terms of 
the Interactive impact on NIE of two aspects of the educational R/D&I: 
the "regional" ediicational R&D labs and the political environment of NIE* 

First, in the mid 1960* s, twenty educational R&D labs were established 
by the Office of Edticatlon (OB) under congressional legislation. By 
the mid 1970* s, only eight of these labs remained, and their regional 
orientation had been lost to a very great extent. Currently NIE has 
responsibility for (and allocates a significant portion of its budget 
to) these remaining labs, even though they are autonomous organizations 
and are not technically a program of NIE. 

Second, NIE*s re-authorlzlng legislation specifies that a significant 
portion of NIE*s budget be used to Insure thau the educational R&D needs 
of all regions of the country are met. The intent of the legislation 
appears to mean support for regional educational R&D labs, and the legis- 
lation has been so Interpreted by the National Council on Educational 
Research (NCER Resolution 18). 
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At t]ie outset of this analysis, we noted a lack of clear fnd conanon 
understanding about regionalism per se: nature and meaning of region- 
alism; the factors and dynamics which most critically Impact regionalism; 
the contextual forces which push for or against regionalism or particular 
kinds 'Of regional approaches; the nature and Implications of alterna- 
tive ways of conceptualising, and designing for regionalism. Thus, we 
chose tp attempt to understand regionalism in ways that would be helpful 
to policy makers. In so doing, we found we were, in effect, breaking 
much new ground. 

To develop such an understanding of regionalism we chose first to examine 
the context for regionalism. Thus, the analysis first ovexviews both the 
educational ft/O&I context and the federal context (since NXE is a federal 
agency) . As a next step , the analysis develops an tiderstandlng of con- 
ceptual and operational aspects of regionalism. The third step was to 
look at regionalism in relation to the various ^O&I functions. T!ie 
final step, then, was to ask how these various aspects the regional- 
ism issue converge and interact In terns of designing for regionalism 
from the perspective of a iolsslon-oriented agency such as NIE. 

\ • . 

5. A Contextual Approach toyProgram Planning 
September 1977 

Unlike the rest of the materials in this collection, this piece does 
not provide a completed policy study. For the reasons to be later 
described In the preface to this paper the project could not be carried 
out as planned. However, the Introductory section had been written 
prior to the aborting of the project and this contained some con.:ept8 
which added an Important dimension to our work. As such it was submitted 
to NIE as an interesting thlnk-piece, and is therefore Included in this 
spirit as part of the total collection of policy studies. 
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The paper focuses on the political context of the program planning process, 
but recognizing that NIE functions at the intersection of the political 
and scientific systems of which It Is a part. As such. It elaborates on 
a point aade In the Agency /Field Relations study In which we pointed out 
the need for NIE to take Into account not only programmatic outputs, but 
also Impact on the R/D&I system and its constituency, in its decision 
making. 

The paper goes on to focus on the Inadequacies of process and rational- 
systems frameworks for program planning and to make the case for the 
building In of political considerations In planning, namely such requlren 
meats as the need "to set Is flee", to base policy making on Incremental 
steps, etc. This then becomes Integrated Into system-wide considerations, 
which were to have Included (but have not been developed in this paper) 
the Implications for all the R/D&I system features (funding, personnel, 
research, development, dlsse^natlon, and so on). This leads to the 
recommendation that program planning should be conducted within a two 
dimensional framework, at the program level (In terms of values to 
NIE stakeholders: political system building) and at the project level 
(across the functional features of educational R/D&I: R/D&I system 
building). Finally, some considerations for the monitoring require- 
ments to be generated are discussed. •« 

6. R&D Coordination In the Social Science Context 
November 1977 



This analysis Is a suomtarlzatlon (In modified form) of a paper pre- 
sented at the Conference on Social Research Organizations at the 
University of Pittsburgh on October 20-22, 1977. 

Coordination is a critical issue from a number of perspectives; 
Among the various R/D&l functions; between knowledge producers and 
knowledge users; among the various institutlo:*^ and personnel and 
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aerosft the various progcana within a specific R/D&I function such as 
research; in terms of R/D&I system maturation and development; between 
funders and the "field"; and so 6n. Further, in the social science 
context, R&D coordination is especially problematic. 

/ 

In ihis policy analysis, we view coordination from, a broad rather than 
a narrow understanding of the concept of coordination — • a broad under- 
standing which is not United to issues of timing, resource allocation 
and Integration in selection to specific programs, projects and related 
personnel activities. Rather, our understanding of coordination is one 
v^.ieh focuses on the nature and needs of a total process of Innovation, 
which considers the meaning of coordination In relation to a total 
process of InnovatlozK' to an R/D&I system of which specific organizations 
and their programs, etc., are a part; in relation to the larger con- * 
text within which the R/D&I system and its organizations, programs and 
personnel exist and with which they Interact; in relation to R/D&I 
system needs and purposes as well as the needs and purposes of organi- 
zations aod their programs. 

We have In this analysis atteo^ted first to gain an understand- 
ing of the context of social science R&D can impact and he im- 
pacted by social science R&D coordination. Thus, we have in this 
analysis raised issues of R&D system maturation, emergent process 
of coordination, lead roles and agencies, and the nature of 
problems associated with the purposes social science R&D coordina- 
tion might be intended to serve. These are, we believe, the, type 
of issues which are critical for R&D coordination in the social 
science context* 

7, Analysis^ Selection and Planning of Programs and Projects by 
the Division of Industrial Energy Conservation of the Energy 
R&D Adtninistration; Phase One Report 
September 1977 

The Industrial Energy Conservation Division (INDUS) of the Energy 
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Research, and Development Administration (ERDA) has a mission which 
Is broad In scope; requires consideration of many complex factors; 
must <^ten be accosqpllshed under conditions of high uncertainty 
or risk; and may Involve conflicting governmental goals. Further, 
consideration must be. given. to the fact that INDUS mpst accomplish 
Its mission as a "lead agency" ' among many autonomous Institutions ^ 
(Industries) which have a large degree of ultimate control over the 
accompl i s h me n t ot INDUS *s mission. Further yet, since INDUS Is 
a funding agency, there will be a '^multitude of voices" besetting 
and beseeching the Agency for funding. 

Thus, It Is H^eratlve that INDUS have a pYocess for the analysis, 
selection and planning of programs and projects which; 

I 

j 1) permits analysis, selection and planning to be grounded 
/ In. a cod^rehenslve knowledge of the broad range of relevant 
system and en^rlronmental factors; 

I 

2) at the same time, permits Identification of those factors 
which are most critical and/or about which cxirrent Infor- 
mation Is Inadequate; 

3) takes Into consideration not only the knowledge production 
Issues of R&D butt also the "downstream" knowledge utili- 
zation Issues and lltltage Issues of need Identification, 
dissemination (Including marketing, distribution, diffusion) 
and evaluation; 

4) takes luto consl legation the nature and* dynamics of the 
relationship between INDIES as a funding, lead agency and 
the "field" of knowledge producers and knowledge users; 

5) takes Into consideration both long and short term needs, 
dynamics and program/project implications; 
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6) permits orchestration and synergy across programs and 
projects. 

This report » theh, focuses on the development. In a manageable and 
useful format » of such an analysis* selection and planning process* 
The process suggested builds upon but extends current INDUS processes. 
It distinguishes between mission areas, programs and projects; thus 
projects are not considered In Isolation but In terms of "port- 
folios'* (l*e., programs). It provides both for organizational memory 
and for monitoring. 

This report Is a "phase one" report. It provides a basic out- 
line of an analysis » selection sad planning system In the form of 
flow charts and of specific questions to be raised at various 
stages of the selection and planning process. These are tenta- 
tive and will require considerable Interaction with INDUS per- 
Sonne 1 In order for the system design to be "tailored" to the^ 
specific context and needs of INDUS. This will be the focus of 
"phase two" of this project. 



8. A Contextual ^_proach to Developnient and the Role of 
Technology In Developing Countries 
September 1977 

The subject of the role of technology In development of LDCs 
(Less Developed Countries) has received a great deal of atten- 
tion in economic, science policy, R&D nanagenent and Inno- 
vat Ion literatures* To date, how<»ver, our knowledge Is 
fragmented and often conflicting. Two of the prime bodies of 
the literature are focused on questions of: 



<1) "Appropriate TechnolWy" — which Is concerner] with 
. vh»t kinds of technologies are appropriate to LDCs 

vlth their low capltalNand high unskilled labor 

' * * \ 

availabilities as compared to the converse for the 

vestem nations^ which areXsources of most 
technology; with the Implications that the techno- 
logies which have been exported from the advanced 
nations to the LDCs have not been appropriate; and 

(2) "Dependency Theory" — which crltlci«es the role of the 
vestem nations In third world countVles (In terms of 
their having denuded LDCs of capital ^toek and of having 
replaced political colonialism with econ<»iic and 
technology based control)* 

These two perspectives find little integration in the existing 
literature. Nor do we find much to guide us in developing a 
comprehensive perspective as to the conditions that are determinate 
of appropriateness. Nor are we presented with any entry points to 
break into the dilemma between the desire to avoid dependency and 
the need to benefit from the sources of most technology th^^reside 
in the West — usually within multi-national corporations (MNCs). 

There was clearly, a need to have a framework of analysis that could 
identify the rich complex of variables (political, economic, socUl, 
cultural and technological) that needed to be considered, and 
within which the tensions could be resolved. Besides being able 
to say what really did make & given technology more or less 
appropriate or what did/should determine the choice of techniques, 
we needed to be able to go further and identify "appropriate 
products," "appropriate R&D systems," etc. Clearly, the problems 
called for contextual analysis. In this work we are embedding the 
existing bodies of theory and current political Issues (national * 
and» international) in our analytical scheme. 
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PREFACE 



THE C(«TEXT OP THE IS^UE 

Bftfora beginning our analysis of the appropriate balance be* 
twaen "Fleld-Inltlated and Agency-^^rected" R&D^ It will be Ixnoor-, 
tantsto frame the issue in its proper context. This can be Intro- 
duced^N^ brief 'discussion ox some central questions for HIE today. 

What dre NIE's responsibilities towards U.S. education in 
general ani educational R&D in particular? How can the Agency 
utilise every limited budget for the best short and long term 
effect t aud^biftamight it Justify a proper Increase in this budget? 
These are central policy questions > as are such related issues as: 
the number and types of personne? "equired within the Agency; the 
proper place of an in-house Besearch effort (if any) ; how NIE 
should relate to the "Field"; the current priorities for NI^ in 
the "Field"; an4 NXE*8 relations with other Agencies that may be 
playing roles in either educational R6J) or in educational practice. 

Any attempt to deal with these Issues must begin with a res-* 
^nltidA of their inter-connectedness. For all its looseness, its 
inadequacies'^ there Is an educational R£J} system to be dealt with 
and NIE is a critical element of that system. Vhat is done in one 
area or in relation to one issue, will likely affect other areas 
and have impact on other Issues. For example, the buil^-up of 
development efforts in the past has had Implications for the state 
of Applied Research and the needs and opportunities for Dissemina- 
tion now. What the Agency adopts as its mission will determine its 
budgetary priorities and lA turn how it should and could relate to 
the Field. 



The recognition of the current state of educational R&D (in- 
cluding the total Innovation process) is as critical as the ree- 



ls a very "immature" and weak R&D system. What then are the im- 
plications of these two characteristics of being a system and being 
imnature and weak? 




ognition of its systemic character. It is loose; gaps are 
characteristic; and inadequacies are all too cjpmmon. In short. It 



11 



X£ education Is to^Se servedHby a quality R&D system, two major ^ 
requirements will need to be satisfied. These Involve (a) system build- 
ing, maintenance and protection and (b) system orchestration. 

Only futility and frustration can come from potlbcles that Ignore the 
state of the educatlox^l RfidO system; policies which Implicitly assume: 
viable Research/Development/Dlssemlnatlon and other Institutions which 
are reasonably well linked. to each other and to practice; policies which 
assume that users are able to adopt quality R&D outputs, able to. properly 
generate and implement their own significant innovations and able to iden- 
tify 'and feed forward their real needs to Developers, etc.; policies which 
assume that the provision of funds to procure R6J) outputs and progrsms are 
the primary requirement for success. We suggest, in contrast, that system 
building, institution bulldlxi^ and rebuilding, and personnel development 
are top current priorities for educational RfiO). Further, it is not enough 
. " merely to build* A fragile, politically exposed and weak system must be 
maintained and protected. 

A mature B&D system orchestrates itself. Relationships are well de- 
\ veloped* Participants know whan to seek and to deliver, from and to where, 
^ and what to expect and trust. An immature system needs help to grow, to ' 
learn how to achieve such a self organizing state. 

These are the needs. NIE may not be the largest governmental contrib- 
utor t:o the educational R&D establishment, but it Is the lead, the core'^dls- 
cretionary agency. NIE must take responsibility for the system building 
and orchestrating. No other body can or will. NIE could walk away or be 
prevented, politically, from assvmting this mission; but the need would 
remain, as well as the ensuing frustration. In our analysis we have taken 
the assumption of thl" mission as a given, along with our premise that 
there is an educational R&D system, albeit immature and weak. 

This sets the scene for our analysis. If we are to understand the 
factors that determine the what, when, how much and how of the "Field - 
Initiated vs Agency-Directed" R&D issue, it must be in terms >f our under- 
standing of the total educational R&D 8y»cem as it now is; as we might 
wish it to become; and as it varies across the differing elements of the 
system (e.|i. ad between Research and Development). To do this we need an 
appreciation of what is fundemental and generic in R&D and what is charac- 
teristic of the present educational R&D context. 

O 6 * 
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Such understanding will need development and presentation If 
ve are to be able to deal with the Issue at hand. It may, Incl- 
dently (because o£ the aforementioned Inter-connectedness o£ 
systea issues) also provide some insight to the type of questions 
raised at the start of this preface. Some very brief thoughts on 
those questions may be a fitting introduction and entree into our 
analysis. 

Satm RETATED ISSUES 

The limited sise of MIE's budget in relation to the needs^ and 
even in comparison to vhat is being spent In total on educational 
K&Dp is veil recognized In the Agency and In the National Council 
on Educational Research* What are some insights that may be help* 
ful in developing parameters and guidelines for budgetary planning 
that may be derived from the analysis that we present below? 

This is clearly a large policy Issue and we can only hope to 
suggest s<»ne useful perspectives in these few comments • 

We would ne^d to begin with an evaluation of the capacity of 
the R6J> system^ overall and across its parts (or functions as we 
will be terming them^-Basic Research^ Problem-Focused Research^ 
Development> Dissemination^ etc*)* What is there now (in terms of 
capacity ^ product inventories ^ etc*)? What can be delivered? What 
Is needed (capacity > outputs) now and over the longer term? In our 
analysis we will point out the needs and requirements of the diff- 
erent function8-»-*and how these may differ over time* 

Specifically^ in our analysis we will note that because of 
funding policies in relation to the Development ranction^ there are 
some quality Development organizations and an inventory of Develop-* 
ment productct* What has been missing have been the quality control 
^ function^ the Developer /User linkage and the User/Product matching 
(and tailoring) functions* We will further note In our analysis 
that the Problem-Focused Research and^(until recently) the Dis- 
semination functions have not begn well developed and are weak. 
We will note that overall Zhe educational R6eD system is relatively 
ixmnature * 



To Illustrate how these budgetary parameters might- be applied, 
let us consider what a "balanced" funding procuss for Research. 
Development and Dissemination might be, given the existing state' of 
these system functions. 



As we ^oted, a relatively large amount of funding has been pro- 
vided in the past for Development, while Problem-Focused Research 
and (until ^recently) Dissemination has been relatively less developed. 
Thus, a "balanced" budget plan might be: 

!• Since there are now a sizeable number of Development 



, Ing— >to that minimal level necessary to maintain the 
existing high quality centers. 
2. Since Dissemination Las ItiUBtk 8o*'§ceatly fragnent^* 

ed, direct significant funding to Dissemination— but 
Btot so much as to build a system that would overwhelm 
Users. At this time, quality control, sorting and 
technical service would likely need to be a part of 
the function. 

3* Since la^rge scale iProblem-Focusad Research has been neg- 
lected, provide major funding here*for system building— 
but not at a level greater than the capacity of the 
function to abat^rb productively^ 

4. Provide moderate funding for Ba8l9 Research, for long- 
term 8ystem-bulld5.ng purposes. 



^ A funding strategy such as the above would, of course, have to 
consider existing funding realities, political conditions and the 
particular curren^. needs of Users. 

Additional considerations will be Important for such a "system 
building" budget and policjy orientation . For example: 

1. There mu^t be funding stability over time . System 



building is a sustained rather than an "in-and-out" 
process. A three to five year period would be minimal 
for any kind of system building— and would be com- 
pletely inadequate in the Research function. For total 
system building, a much longer time frame is required. 




products available, reduce current Development fuusd- 
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2* Systeai building is different €rom procurentent of a product » 
and this fact has signlflcitnt implications for funding 
policies* For the procurement of a product, open competi- 
tive bidding is often a systemlcally valid strategy because the 
"product (not an Institution) is the concern. In system ballding 
ing, the reyerlse holds true— > the Ins tit* tlonal (and personnel)' 
base is the priiitary concern—not a product. 

3« From the political point of view it may be vital to attempt to 
educate the releyant communities as to the state of the edu- 
cational R&D system and to the fact that the next few years 
have to be seen as a period of long term capital investment**- 
if we are not to be burdened in the future with the errors of 
the past as we seem to be today. Perhaps this is the only 
meaningful justification that can be used for added Agency 
funding that will not return tO' plague educational RfiJD in the 
• near future. 

I 

Thifese last points warrant further ponsideration. As we will note 
the analysis, system building requires continuity «nd concentration . Di- 
rection and orchestration must be provided from some system-wlde agency 
such as NIE. Thus, Agency efforts cannot be scattered and non-direct ive 
and still ue effective. This, in turn, implies a tighter degree of selec- 
tivity and control by the Agency than would be possible under standard RFP 
and competitive bidding procedures. Thus, there is a dllemma--there are 
legal and political constraints involved in a "sole source" approach (which 
would be a relevant mode of funding for system building purposes). 

Thus, as our analysis will indicate, it becomes important for NIE to 
know what Is needed, to be aware of the legal/political constraints, and to 
find ways to mediate the tension between needs and consi:raint8*<«*l.e. , meet- 
ing the needs without violating the constraints. Perhaps one mode of such 
mediation would be for NIE, acting in consort with other agencies facing 
the same tension, to apprise the Congress and other relevant groups of the 
iong--<'erm xequ.Lreraent8 for system building as contrasted to procurement 
approaches to funding, given the current system state and needs. 

In the above discussion, ve have touched upon twoxOther issue" relevant 
to NIE Jliich are discussed in our analysis: Inter -agency coordination and 
Internal NIE staffing (especially the question of an internal NIE Research 
capability ) . 
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Although a detailed analysis and discussion would be beyond our 
current scope of effort, our analysis Indicates that friter-agjencv 
coordination and orchestration is a key NIE role— both because NIE Is 
the lead agency in educdtibiul R&D and becausa^more funding of educa- 
tional RU) is provided outside of NIE than by tiXE. As we noted 
earlier, this may at times nean that a significant portion of NIE 
efforts may be applied toward a specific critical area eveu though 
relatively little o^ NIE*» budget is applied to this some area. Stated 
.another way, NIE*s focus of concern should be with needs, not only with 
the implications of its budget per se. 

Our analysis also suggests thet the way NIE provides for its own 
internal staffing will have a critical effect on the direction and 
, effectiven^ . of the Institute. For example, the NIE role of orches- 
tration requires personnel who have skills in orchestration and in 
facilitating collaboration between people and/or between institu- 
tions and agencies. Additionally, NIE will need some personnel who 
. h^ve "political savvy" 4 For another example, our analysis will 
suggest >a need to build the Research function and will further sug- 
gest that only by having an internal Research capability will NIE be 
able to orchestrate the building of the Research functions • 

We might also comment btlefly on the issue of NIE "ru'*.es of 
thunib" (such as: "only offer a grants competition when a total of 
one million dollars can be provided and when you can fund 257. of the 
proposals submitted"). We would not be concerned with the amount 
and the percentage figures per se. Rather we would note that there is 
an inherent system logic in such a rule of thumb- -i.e., it is correct 
that th^ expectations of the field should not be raised beyond 
reasonable levels of potential for fulfillment. We would further 
note, however, that "rules of thumb" tend, too oft<'n to fall Into the 
trap of ignoring critical system or function dynamics, conditions, 
needs and requirements. In the above case the danger would be that 
such a grants competition would be used In an area so lacking in 
Excellence that the funding of 25% of proposals at the one million* 
dollar total level would tend to trap the 'Agency Into Indeed provid- 
ing the funds to low quallLy, low -success -probability projects. 
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One final cooment. Our analysis may at first glance appear overly- 
extended for such a "simple" problem as 'the Fleld-Xnltlated vs. Agency- 
Directed Issue. Our point Is precisely that this Is not a simple question- 
It Is embedded within other more fundamental system Issues, and NIE's re- 
sponse will have system-wide Impact. Thus, the only appropriate analysis is 
a system analysis. Additionally, this ana! sis has shed light on other 
issues confronting NIE— a good illustration of the "multi-purpose effects" • 
concept we will introduce tn this report. 

REPORT ORGANIZATION AND PREPABATION 

In the pages that follow, our report will: 



1. 
2. 



3. 



4. 



describe our method of analysis; 

analyze the NIE/ field relationship issue from both the generic 
and educational context perspectives in four major functional 
areas of educational R/D&I— Research, Development, Dissemlna- 
tionr'^d Evaluat|.on Research; 

outline potential In^lications of the analysis for NIE pol- 
icies and strategies in each of the four functional areas; 
provide hypothetical scenarios to illustrate the implications 
of potential policy and strategy decisions of NIE. 



The policy analysis Is framed within the overall contextual issue 
of the relationship between a mission-oriented federal agency (NIE) 
and the operative R/D&I community in a specific sectoral context (edu- 
cation in the United States). The specific issue of Fie Id- Initiated 
vs. Agency-Directed Procurement is treated as a sub- issue of the larger 
Agency/Field relationship Issue. 

Reference 

Selber, Sam D., "The Requirements of a National Educational R&D System" 
in Educational Researcher, December 1975 (p 3-6), 
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This policy analysis has beeu prepared by staff members of the 
Norchvestem University Center for the Interdisciplinary Study of 
Science and Technology under contract i' NI£ -0-400-76-0 110. 

The Center is an interdisciplinary Research center that exists 
'to focus the broad 'and varied interests of a number of Northwestern. 
University faculty concerned with the moral, philosophical, cultural* 
social, economic, and political significance of science and technology. 
An area of special concentration is on R§D nanazement systems and 



dustry, law enforcement, and education} . The Center brings together / 
interdisciplinary talents from such diverse fields as management, 
economics, history, philosophy,' education, journalism, the physical and 
biological sciences, engineering, psychology, sociology, anthropology, 
and religion. Faculty and staff associated with the Center are drawn 
'from several organizational units of the University, including personnel 
with appointments in the Graduate School of Management, the School of . 
Education, the College of Arts and Sciences, the Technological Insti- 
tute and others^ Center Associates are affiliated with universities 
and Research institutes across the country and in a number of other 
nations; 
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DEFINITION OF I SSUES AND METHOD OF ANALYSIS , 

I « ■' * 

I.- ISSUE DEFINITION 

1. Initial NIE Concerns 

As presented to us by NIE, thQ concerns of the Agency focused on 
two questions of prbfcurement policy. Foremost was the question of 
determining the appropriate balance between "Field-Initiated" versus 
"NIE-Directed" R5D. The Agency had been receiving considerable 
criticism from the field, reflecting the view tlmt too limited a role 
was permitted the field in the design of either broad programs or 
specific, procurements. Many of those unhappy with the present charac- 
ter of procurements were suggesting that the Institute commit itself 
^ to a predetermined funding percentage for field-initiated RQD. The 
expectation was that such a predetermined set-aside formula would 
create a larger flow of funds into field-initiated work than had recently 
been the case. The National Council in ^ducational Research, in its 
desire 1^ be responsive to the field, Wquested a review of NIE procure- 
ment policies and some careful consideration of the question of how 
Agency policy might be developed on the matter of the appropriate 
balance between Field-Initiated and NIE-Dirocted R5D. 

A second, related question involved determining the appropriate 
mechanisms' for procuring Field-Initiated and NIE-Directed work. Of 
particular concern were questions about the nature and extent of 
direction that an agency could justify building into procurement 
mechanisms, specifically with the requirements and language of RFPs 
(as these might constrain project conception, design, execution, etc») 
and most particularly whether the RFP should be used at all in the 
R^D funding context. 

2. Our Reformulation of the Issue 

'in thinking about these questions, we were struck by the rather 
fundamental and broad-ranging implications of the questions raised, 
especially when viewed from the framework of our understanding of R5D 
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systems and processes. In order to deal with these far-reaching issues 
in a meaningful way, it seemed to us that there were some critical 
prior questions to be addressed and clarified. Therefore, we proceeded 
to reformulate the questions of concern to the Institute in a manner 
that we felt would best permit us to shed some light and suggest some 
directions for policy development. 

Our consideration began With recognition of two defining features 
of the Agency's character: first, that NIE is a mission-oriented RSD 
agency; and second, that NIE is the lead agency for federal activity 
with respect to Research and Osvelopment in education. Given that 
role, its funding policies would have to be understood in terms of its 
purposes as'th^se impact on the total education sector's Research, 
Development and Innovation (R/D§I) system. What the balance of differ- 
ent types of funding should be and how an agency should relate to the 
field with which it worked would depend on the purposes the agency was 
tr>-ing to achieve across all aspects of the R/D$I system. The nature 
of this behavior would need to be fitted to whatever it was necessary 
for NIE to do if it were to achieve its' mission in all its R/D§I system 
aspects. Percentage of field-initiated programs and type of procure- 
ment mechanism used could be viewed in this light as indicators or as 
symptoms of Agency/Field behavior rather than as direct policy lever- 
age points. Therefore, it seemed to us, our analysis could be focused 
most fruitl(ully on more fundamental questions concerning NIE's mission 
and purposes in relation to the field's needs and conditions. Answers 
to these prior questions, if seen as determinants o^ necessary Agency 
behavior, would suggest what the appropriate FielU-Initiated/Agency- 
Direct6d balances should be at any particular time, and would suggest, 
too, the most apjiropriate procurement mechanisms for each individual 
case. With this set of assumptions as our starting point, we then 
began to formulate our analytical strategy. 

3. The Relat ionship of this Analysis to Current NIE Analyses of this 

Issue A ' 

In carrying out our analysis the results of vcont in-house NIK 

6:, u 



4 



efforts (specifically the Duffy et a], 1976 memos) were made available 
to us. We deem it appropriate to point out the areas of similarity 
and points of departure between our. analysis and the NIE in-house • 
report. 

A. Review and Interpretation of the Duffy Report 

From our perspective, the NIE study was a rich, on-target 
discussion but one that was acutely limited by its narrowness of f'^cus. 
We make this criticism on two grounds. 

a) The report lacks an overall R/D^I systems perspective, 
casting its arguments in terms most relevant to the Research function, 
but far less Ij^ropr^ate to the NIE purposes wi«h respect to, and the 
generic issues inherent in, such other functions as Development, ' 
Dissemination, etc. 

b) The discussion was,. in our view, more than necessarily 
couched in terms (of a Field/Agency dichotomy (an "us" versus "them" 
peT?pecti\-'e) . 'Raiher, it will, as we will show, be vital to recognize 
NIE*s integral place as a part of the educational R/DSI system. Despite 
these criticisms we would bf remiss if we failed to comment on the 
quality of the analysis wit In the above stated constraints. 

B. Our Approach 

By contrast, our 'approach grows out of and builds on a systems 
perspective, with NIE's mission being viewed in terms of its impact, 
as an integral part of the system, on the educational R/D§1 system's 
health, functioning and outputs. Further, growing 91't of this systems 
perspective, and as is inherent in our general analytical method, we 
engage in a broader, more systematic analysis ^f R/D^I functions and 
the range of conditions effecting the system. Finally, we note that 
procurement is but one of the range of behaviors available to NIE by , 
which it can influence the system and that behavior roust be evaluated 
in its totality. 

♦ 

Thus, the essence of the issue as wc see it is: how does NIE 

• 

achieve its purposes through procurements and other Agency actions » 
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taken in consort with and a s part of the field? / 

4. NIE Purposes 

A. The Multiplicity of Purpo^s 

Central in determing NIE«s pf^oper modes of behavior must be its 
mission in relation to the educational R/D§I system. While NIB can be 
conceived as seeking many individual goals these can b<3 usefully grouped 
under the following general sy is terns dimensions: 

1. Substantive outputs of the R/D^I system (knowledge, products, 
services 1 etc.) - The systems throughput dimension. 

2. System capacity building (institutions, linkages, personnel, 
etc.) - The level of maturity and capability of the system 
itself. 

3. Affecting the system environment (support, prestige, legiti- 
macy, etc.)- The system environment. 

Procurements tend to be thought of primarily in terms of the 
first of these categories C the direct purchase of R/D§I activities to 
generate knowledge, produce programs, products, etc., or to provide 
services. Occasionally, . gencies procure capability-building activi- 
ties directly, as in the provision of institutional support, or the 
funding of training programs or graduate or post-doctoral fellowships. 
But for. the most part, procurements arc designed and managed by 
agency personnel as individual projects or programs designed to pro- 
duce specific outputs for the use of the operational system or the 

R/D§I system itself. 

\Vhat tends to be overlooked is the extent to which these mani- 
festly single-purpo«5e procurements tend to have multi-purpose impli- 
cations: in almost every procurement (or other Agency behavior), more 
than one of these purposes will be involved, whether implicitly or 
explicitly. Thus, the award of a grant to an R5D institution to 
support a specific project may also have an impact on that Institution's 
capacity to perform in the future (e.g. by permitting it to hire 



additional personnel, by the added experience that may result), 
^iailarly, the provision of an institutional support grant may result 
in the conduct of R^D programs whose outputs may not have been speci- 
fically sought but which ate of considerable value, and at the same 
time act so as to increase that institution's legitimacy vis vis 
various of its publics. 

Consequently, it be^^.omes essential for an agency to be very clear 
about its purposes, those entailing system building and affecting the 
system environment as well as the use of system capacity -.to produce 
substantive outputs. And too, it seems important to develop some 
recognition of the legitimacy of latent as well as manifest purposes 
for j^ocurements as well as other Agency actions. 

B'. Manifes/ and Latent Purposes 

The legitinLcy of late.it as well as manifest purposes of Agency 
actions is a poinf: that merits some elaboration. The manifest reason 
for supporting a particular project may l\ave little relevance to the 
real reason, which is latent, implicit, and infrequently made clear 
to members of the R^D community and/or relevant publics. A particular 
project may receive funding npt so much because of the immediate 
payoff expected from the project itself but rather because of the support 
it is providing for a certain type or group of graduate students, or 
because it is expected that if a certain Researcher is suppdrted long 
enough he is bound ^o make very substantial contributions to the field. 
In such cases, defending a project in terms of its manifest purpose 
may be difficult, but justifying it in terms of long-term capability- 
building needs may be much less of a problem. Or to consider a some- 
what different example, an agency may be subjected to considerable 
pressure to sup/ort a particular kind of pr6gram, and the pressure 
may be sulTstantial enough to have serious enough ramifications to 
jeopardize achievement of important objectives. In such a case, an 
agency may have little interest in the manifest purpose of a project, 
but aay.support it for the latent purpose of r e jficvTn^iindue stress 
on the system. 

r 
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The essential point here is that procurements may provijje the 
greatest long-range payoff if they arc designed with multi-puk-poses m 
mini, and if Agency personnel can design them creatively to s^rve 
latent as well as manifest purposes. IVhat would seem to be nei^ded, then, 
are deliberate Agency strategies to capitalize on the raultiplicjity of 
consequences from specific Agency actions, to maximize possible gains 
and minimize possible costs from potential multiple and interaction 
effects across the latent, and manifest purposes of given procurements. 

• C. Interactio n Effects 

This issue of interaction effects is one of the most critical 
points that seems to be overlooked in the development of Agency policies. 
Once an Agency comes to view its beh, ^or in terms of interactions 
aroong seemingly discrete actions, an entirely different kind of 
understanding emetges of the potentially far-reaching systemic impli- 
cations of individual decisions and policies." Different purposes can 
interact with one another (a point we shall return to shortly) . Pur- 
poses can interact with procurement mechanisms — e.g., a mechanism ^ 
used to procure Basic Research outputs can have major implications 
for long-term capacity-building. Purposes and mechanisms can interact 
with, contextual conditions^ e.g., the state of development of the 
system; a strategy that may have been ineffective a decade ago may be 
highly successful in achieving certain purposes now or ten years from 
now. 

The point is perhaps made most clearly by examining potential 
interaction effects among purposes, both within a single procurement 
and across the totality of procurements made by an agency. A procure- 
ment can lead to the creation of outputs and lead to an iipnrovoment 
in the system environment. Or, it can lead to a deter io rat', n in the 
environment if, for example, that particular output is seen as 
offensive to certain key elements. It could also luad to a destruction 
of R/ntil system capacity by, for example, moving critical resources 
away from their most productive areas of application. 

IVhen one examines patterns of Agency action*; across procurcmcnt.s -- 
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i.e., when one considers potential interactions among the discrete 
procurements that make up an agency's "portfolio" — interactions of 
an even loss obvious nature become apparent. Across programs, the 
outputs nay reinforce each other (synergistic effects). Or they may 
counteract each other in the manner of what might be called "anti- 
purposes" — i.e., taking a specific action in pursuit of one purpose 
nay make more difficult the achievement of another purpose. The use 
of RFPs to procure certain kinds of Research, for instance, might 
well have anti-purpose effects if a by-product is turning off the 
best Research talents, suggesting to them that Research funding in 
the field of education is unlikely to be forthcoming without untenable 
constraints. Such e:^cts may be immediate in their interaction or 
observable only in lagged and in second- and third-order manifestations « 
If an agency decides to design procurements that are deliberately 
multi-purpose in nature, it becomes essential for agency personnel to 
have a clear understanding of the kinds of procurement "adu-ons" that 
tend cO be congruent vs. incongruont with each other, functional vs. 
dysfunctional . 

Portfolio effects may be discernible within institutions as well 
as across institutions. It is common to observe how R^D institutions 
become shaped by the patterns of funding that become available to them. 
If a single agency provides a particularly large share of an organiza- 
tion's total funding, agency actions can hllve the effect of molding 
br changing the very character of such organizations. 

In summar , then, interaction effects will need to be considered 
in terms of their: 

- synergistic effects 

- congruency/incongruency with each other 

- lagged (and indirect/second and third-order) effects 

- cumulative effects within and on institutions and personnel 

Tho essential point is that multi-purpose effects are inevitable. 
The issue is not whether there should be mult i -purposes but rather 



9 



;»hether they arc to be recognized or ignored, and if recognized to be 
dealt with and capitalized upon or anti -purposes minimized. 

5, The Functional Context of NIE Purposos 

Up to this point, we have considered three sets of purposes that 
can be affected by procurements and other Agency behaviors. We turn 
now to consideration of the second major building block of our analyti- 
cal approach. 

NIE*s purposes are achieved through the carrying out of various 
activities that can be categorized by R/D§I functions: Research, 
Development, Dissemination, etc. Achieving the same purposes (e.g., 
institution building or affecting the system environment) may call for 
• different Agency behaviors in relation to these different functions. 
Building Research institutions may demand strategies very different 
from those required to build Development organizations. The mechanisms 
that are appropriate for procuring Development products may be quite 
inappropriate for procuring Basic Research studies, and so forth. 
Similarly, the types of skills and experience required within NIE 
to work with personnel and institutions involved in Basic Research are 
likely to be rather different from those required when working with 
those involved in the Dissemination function. Therefore, it follows 
that the determination of Agency behaviors (of all kinds) are likely 
to be highly dependent on the R/D§I functions with which they are 
involved, and this consideration must be reflqgted in our policy 
analysis. 

To make such analysis posstble, it was necessary for us to select 
a set of R/D5I functions that seemed to be reflective of the bulk of 
the activities that go on within educational R/Df|I and that become the 
ob)0.^.t of NTH procurement and other Agency activities. With this in 
mind, wc selected the following R/D^I functions (or groupings of 
functions) to become the focus of our analysis.* 



♦ In the R/D^l systems analysis scheme wc use generally, we treat Need 
Identification as a discrete function. In education, however, whore 
specialized Need Identification mechanisms tend to be lacking, Need 
Identification is carried out as an integral part of each function. Wc 
havo therefore treated Need Identification this way in our policy analysis. 
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A. Resoarch 

Research can vary along a scries of multi -dimensional continua, 
generally categorized as going from Dasic to Applied. We fully 
recogr.i::e the debate involved in such categorization and the diffi- 
culties involved in the usual over -simplification so implied. Never- 
theless there are, for our purposes in relation to discussing appropriate 
Agency behavior, important potential differences between what is 
required to deal with Basic Research, which is largely involved in the 
search for knowledge for its own sake, and what is required for applied 
work that goes on in relation to well-defined problem areas. We will 
therefore examine these two types of Research as being representative 
of the range of activities with which NIE may become involved win-"" 
thinking through the implications of our analyfi«*-for''the particular 
kinds of Research NIE procures. Agency personnel can make the necessary 
accorjaodations to variations encountered between these two extemes. 
The two sub- functions therefore will bo: 

1. Basic Research (the seeking of knowledge for its own sake) 

2. Problem Focused (or Applied) Research . As used here and 
later in this analysis "problem" refers to a social or 
practice-centered problem rather than to the kind of 
intellectual or discipline-based problem that is central 
to Basic Research. 

B. Development 

Whereat what we are calling Problem- Focused Research is oriented 
toward problem areas w thin education, Development work tends to be 
focused on the design aad elaboration of products, processes, programs, 
procedures, practices, etc. that attempt to deal with identified 
problems or needs. I-or simplicity of usage, we will generally use the 
term "products" to describe the outputs of the Oevelopment process. 
However, it should be understood clearly that we have the full array 
of Development outputs in minU ~~ programs, procedures, strategics, 
practices, etc. as well as the narrower category of outputs generally 



thought of as "products." ' 

C. Dissemlnation/Implementation/Utllization 

The functions within this cluster are typically treated separately 
in R/D5I systems analyses (at least as between Dissemination and the 
other two). We will *ink them in this analysis since this is a 
characteristic of the educational R/D§I system: its relatively low 
level of maturity and- the general absence of institutionalized integral 
User change agent functions make this a necessity. The focus of our 
analysis will be placed on the Dissemination functioji, since 5t is here, 
in system-level linkages (rather than within the User setting that 
shapes Implementation/Utilization), that NIE efforts can have more 
substantial impact. However, it should be noted that any future ex- 
pansions of our analysis should Involve specific detailed analysis of 
the Implementation/Utilization functions. 

D, Evaluation/Policy Research 

Evaluation and Policy Research are often grouped together. In 
education this seems particularly fitting since Policy Research in 
education so often involves one or another form of Evaluation Research. 
However, there aie important differences between Evaluation Research 
and Policy, Research, and they may require some extended separate 
treatment. Since it appears that the bulk of NIE«s efforts in these 
areas are more directly involved with Evaluation Research, we will 
focus on this function in our analysis. Again, further work could 
expand on the specific issues related to Policy Research. 

6. R/DSI System Context 

We havo seen up to th^s irt in the analysis that achieving the 
same Agency purposes may require sct»ewhat different Agency behaviors 
in relation to different R/dt,l funcr.ionr.. But beyond this, for R/D(il 
functions and their generic rcquircnenrs to be understood in terms 
that seem congruent with concrete empirical jreaUty. it becomes Im- 
portant to sec each function within a total K/im system context. 
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We define an R/D5t system context as the joint interaction of 
three elements: 

1 . the R/D^I functions , as described above . 

2. operating conditions within a particular R/D^I system 
(e.g., the maturity of the system; the types and quality 
of personnel available; the types and quality of the insti- 
tutions available; the state of development of the know- 
ledge/technology base; the ndture of the information 
systems and flows among key elements in the R/D5I system 
journals, invisible colleges, conferences, etc. etc.; the 
very nature of the innovations involved ) . 

3. the system environment (e.g., the political/social environ- 
ment, especially its supportiveness or lack of support for 
the system; the economic environment; the nature of the 
know J edge base of the field — whether it is a natural- or 
social -science base; etc.). 

Therefore, in determining appropriate Agency behavior for achiev- 
ing a certain purpose it becomes vital to be cognizant of both the 
system function to which it applies and the manner in which the generic 
characteristics of a given function are mediated by systemic and 
environmental conditions. It is this joint effect that we term the 
R/D§I system context. 

7. The Spe.trum of Agency Behavior 
A. Types of Behat^ior 

While an agency such as NIE can engage in a wide variety of be- 
havior in relation to its purposes, these can be usefully grouped for 
analysis under three headings: 

1. Procurements 

An agency's use of the funds available to it. to procure specific 
outputs, institution building, etc. is generally seen as the prime 
forum for Agency action. 



13 



2, Xon-Procurement System Behavior 

The potential range of Agency behaviors is 'not necessarily limited 
to procurements. It may also play a potentially important role in the 
system through a variety of other kinds of actions. These may include ^ 
working with other agencies so as to achieve synergistic and multiplier 
effects from joint (additive and/or supplementary) activities; having 
Its personnel play an active role as members of the R/D5I community— 
as Researchers, as participants in conferences, as influences on 
thinking in various informal interactive modes, etc.; by making in- 
formation available; etc. 

3. NIE Internal Actions 

How an agency relates to a field will be importantly determined 
by what goes on within the agency. Specifically, the strategies and 
modes of behavior required will be constrained by the exten* o which 
NIE has the number and type (skills, experience, stature) of personnel 
needed and the organizational and budgetary structures that permit 
appropriate behaviors. 

• B. Types, of Strategies 

Within the above three areas fcx N.R behavior, there remains a 
wide and multi-dimen^iional variety of behaviors in which an agency 
can engage. These can be ondensed along three dimensions: 

1 . Degree of Ag ency ( ontrol 

NIE might nee the need to maintain a greater or lesser degree 
of control over what goes on in the field, in terms both of extent 
(level) . id of do»r.ain (program selection, methods used, personnel 
involvec', nature of relationships, forms of reporting,etc.) . 

2. Degree of Agency I nvolvement 

might choose to be more or less directly involved in what 
wa^ occurring in the field— tor extimple: NIB participation in the 
various functions (e.g., undertaking various types of Research, 
Dirscmin.ition, Hvaluations, Policy and RSD system studies); planning 
for anu monitoring of the R/Df,I system; dotormining what work to carry 
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out internally and what activities to procure through external con- 
tracts or grants. 

5. A Strategy Continuum 

Seen the above ways, it is possible to array very roughly the 
types of strategies by which an agency can relate to various aspects 
of a field. Such an array could be the followikg: 

- Initiate activity for a field 

> Supplement what is already going on 

- Modify what is already going on 

- Selecjbi^rom among what ^s already in a field 

- Educate the field to operate differently 

- Mediate external pressures on specific field elements or 
programs 

- Integrate programs, institutions and systems to be found 
in the field 

- Cooperate with other programs (e.g, in other agencies) or 
with field programs 

- Facilitate activity already going on 

- Execute activities i^iitiited by the field 

- Evaluate activities that have occurred in the field 

- Monitor what is going on. 

This array, while admittedly rough, represents, jointly, a 
diminution of Agency control and involvement in lat is going on 
in the field, in both pragmatic and systemic terms. The order is 
not, however, especially important. To the extent that it does 
array a variety of possible strategics, it demonstrates a richness; 
that goes well beyond the simple Field-Initiated vs. Agency •♦^Directed 
continuum. From our perspective, the Field-Initiated vs.4 
Agency-Directed continuum is likely to be most meaningful in 
relation to individual projects and programs rather than system- 
based purposes, and in relation to those functions in which programiratic 
activities can be separated ^asily from system activities, at least 
in the short run, as in the case of Research* especially Basic Research. 
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(And even then the extent to which the FIS vs NIE continuum applies 
see=»s related to the extent to which Agency personnel focus on their 
own individual projects ra:.ier than the system as a whole.) Tlie Field- 
Initiated vs. i^IE-Directed continuum is likely to be far less meaning- 
ful for such inherently systemic functions as Dissemination, or for 
system building purposes. Seen in this way a determination of the 
proper proportion of NIE's budget that should go to Field-Initiated 
activity could only be made in relation to NIE*s agenda, as it played 
itself out with respect to the Agency *s purposes, as manifested in 
the various R/DftI functions (Research, Development, Dissemination, etc.), 
under the prevailing contextuaUconditions (personnel, funding, 
maturity of the system etc.). As an overall index it would therefore 
not seem to provide much meaning, and hence not represent a proper 
actionable policy criterion. 

8. A General Analytical Model 

As implied above, our analysis will involve the specification of 
the behavior appropriate to the achievement of Agency purposes in the 
context of the functional, systemic and other environmental conditions 
that prevail. Diagrammatical ly, this can be illustrated as in Figure 
1. This model also indicates that NIE*s purposes will themselves be 
influenced by what is going on in the educational R/D§I context and 
that in turn this context will be importantly influenced by how NIE 
does actually behave, as an integral part of the educational R/D5I 
system. ^ 

II. METHO D OF ANALYSIS 
'--^ ^-3^ 

1. General Methodology 

A complete analysis would require examination of how the inter- 
action of NIO»s purposes , as manifested in the R/D8I functions , and 
mediated by the systemic and environmental conditions, determine 
appropriate Agency behaviors and consequently strategics in relation 
to the field. The above statement would imply at least a four 
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Figure 1. Appropriate NIE Behavior as a Function of NIE Purposes 
and the R/D&I System Context 
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dimensional analysis. As a simplifying step, we have elected to 
consider each of the R/D5I functions separately in relation to the 
combination ,of R/0&I systemic and environmental conditions, as shown 
in Figure 2, 

* » 

' R/D§I Environmental and Systemic Conditions 



R/O^I Function 



(Appropriate NIE Behaviors/ 
NIE Strategies) 



Figure 2. . Simplified Analysis Matrix 



The cells of this matrix are the appropriate NIE behaviors which can be 
built into strategies that the Agency could pursue. 

Having conducted the individual analyses for each R/D&I function, 
it becomes possible to consider the implications for NIE strategy 
across the functions. Finally, the strategies can be converted into 
scenarios in which fpatterns of hypothetical, or actual past or con- 
templated NIE behavior, are analyzed to suggest likely impacts if 
impl«nented as originally formulated, or as reformulated in alternative 
ways that take into account some of the points we have tried to under- 
score in our analysis. In the final analysis, one could conceive of 
converting NIE*s total programmatic agenda into an integrated set of 
scenarios, determining the various patterns of appropriate Agency 
behaviors (a fallout af^ which would be an estimate of the percentage 
of all procurements that would be Field-Initiated--though as we have 
stated this would not be an index of great significance in an of itself), 
and suggesting likely impacts. ^ 

2» Specific Analysis Met hod 

Within the framework of the above general approach, the procedures 
to be followed will be as follows: 



80 



I 



18 



A« Functional Analyses 

Each sel*"'*ted R/D§I function (Research- -Basic and Prob i em- Focused— 
Developnent, Dissemination, Evaluation) will bo analyzed in the follow- ^ 
Ing manner: 

1. Generic Characteristics of the Function 

We will begin Bach functional analysis with a review of those 
generic issues inherent in the function which are likely to be of 
relevance to educational R/D^l and have implications for NIE behavior. 
A complete generic review of each function would inevitably deal with 
many issues that are of relatively lesser concyem in education at this 
time« Gi^n the constraints of time^ volume yand salience implicit in 
a policy analysis^ we have attempted to be jiidicious in our selection 

of issues to be discussed* / 

/ 

2. The Educational Context of the /"Function 



We then consider the current stat^ ^f affairs in the educational 
R/I>QI system as it relates to carrying out this particular function. 
We examine contextual issues pertaining' to the state of development of 
the relevant knowledge base, the institutional base, the personnel 
base, the climate of support fo'r funding its activities, etc. As before, 
the implications for NIE behavior are drawn. Wliere feasible, this 
section concludes with some general guidelines for NIE*s operating modes 
and strategies. •'^ 

3. Implications for Agency Behavior 

In this final section of each functional analysis we attempt to 
summarize the requirements fox NIE behavior in relation to the parti*-' 
cular function in its present context, building up some recommendations 
for Agency strategy. 

ii. Cross-Functional Analysis 

In this critical chapter, we take the analysis and strategy- 
building a step higher by attempting to draw the cross-functional 
implications for NIE at a total Agency level. 
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C« Scenario Analyses » 

Two illustrative scenarios are then analyzed. Each scenario 
consists of a description of a hypothetical NIE behavior 
(e g., a particular procurement^ program, its objectives, the manner 
of its implementation), an analysis of its wide implications (if any) 
and likely impact, our recommendations as to what changes might 
(or should) have been made, and the likely consequence of these 
recommendations . The thinking behind this process is depicted 
graphically in Figure 3. 

A specific NIE action (usually a procurement), with its intended 
purpose, would result in some consequences. The effects or impacts 
would be a conserjuence of the NIE action interacting with contextual 
conditions, and would have to be understood in terms not only of the 
intended and manifest purposes of an action, but also any other 
(possibly latent) purposes. These impacts would be evaluated and 
appropriate strategy alternatives recommended. Such recommendations 
would lead to NIE actions involving procurement, non-procurement and • 
internal NIE behaviors, in relation to the- whole range* of possible 
purposes. Then, in turn, the effects of these behaviors would be, 
analyzed. I 

The scenarios are intended to suggest an analytical approach we 
view as appropriate for internal Agency use in designing procurements 
and relating procurement strategies to other, possible non-procurement 
courses of action. The strength of this approach, we would argue, is 
the manner in which it orients Agency personnel toward system-level 
thinking: 

1. It requires the analyst to think in terms of the 
multiplicity of purposes implicit in procurements and 
other Agency behaviors, anJ sugjiosts the legitimacy of 
designing courses of action in terras of latent as well 
as manifest purposes . 

2. It requirek consiueiat ion of interaction effects among 
purposes, between purposes am. nochanisms, and between 
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purposes and generic charactoristics of R/D5I function^; 
. as these are mediated by systemic and en * ror-ental 
\ conditions • 

^. Nit calls for Agency personnel to estimate potential 

impacts of contemplated courses of action on key dimensions 
ofXthe system. 

4. It ^ggests a series of questions that enable the analyst 
to generate alternative courses of action with potentially 
different impacts, and to use these alternatives, along 
with their own estimates of potential impacts, to reformu- 
, late and refine contemplated strategies. 

Wc offer the analysis which follows as a first -cut at what we 
believe can evolve over time into a highly useful approach. 

REFERE NCES 



Puffy ot al ,, 1976.' ^» 

iVe had access to several internal memos, in various states of their 
devclopi It, in August and September 1976, including: a) an August 2, 
1976 memo written by Susan Duffy and Noel Brennan entitled "Status 
Report on FIS"; and b) a somewhat later set of materials providing 
working definitions of Narrow and Broad FIS, elaborating the 'relevant 
dimensions of each, and considering some of the implications for NIE. 
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RESEARCH 

BASIC KESEARai; KNOWLEDGE FOR ITS OWN SAKE 

? 

I. GENERIC CHA R. \CTERISTICS OF BASIC RESEARCH 

1. Uncertainty and Unpredictability 

2. Time Frame for Outputs 

3. The Rate at which a Basic Research Area can, be Built Up 

4. Some Other Fundamental Issues 

• A. The Criterion of Excellence 

B. Competing Centers of Excellence 

C. "Minimum Critical Mass" 

II. BASIC RESEARCH IN UCATION 

1. Weakness of Central Core of Educational Researchers 

2. General Climate 

3. The Interdisciplinafity of Education 

^. The Funding Sources for Educatioaal Research 

III. ' i;.!? LIGATIONS FOR THE ROLE OF NIE 

1. Procurement Behavior 

A. Facilitate Orchestration 

B. System Building 

C. Implications of Levels of Consensus in a Field 

D. Process Mode of Management 

2. Considerations Internal to NIE and Non-Procurement 
Activities 

A. NIE Staffing 

B. Non- Procurement Activities 

C. Relationship between NIE, Staffing, Non-Procurement 
Act i\ iti.es, anu a Wider Range of Procurement Options 

PROBLEM- FOCUSHD RESIiARCIi 



1. Introduct'on 

2. Users and Fundors: Seeing Problem-Focused Research as 
a Development Activity 

3. Probloms from the Perspective of the Researcher 
4 The User's Uole 

S. Is.sucs Related to the Characteristics of Peronncl In- 
volved in the Problcm-l'ocusod Research Process 
A. Larp,c-Scale Resoatch and the Rcspojise to the 
Recjuirod Inst i tui iojuil Setting 
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B. Redefinition of Studies into a Basic Research Mode 

C. Problems in Maintaining the Interdisciplinary 
Focus 

6. Summary 

II, THE FDUCATIONAL CO^f^EXT OF PROBLEM-FOCUSED RESEARCt l 

1. Basic Weaknesses 

2. Institutional Bases ^ 

A. The Universities 

1. Scaling Down the Problem^ 

2. The Ad-Hoc Approach 

B. Large R5D Organizations in the Private and Quasi- 
Sectors 

3,, Some Additional Considerations 

A. Communications 

B. Commitment 

C. Climate 

D. Need Identification 

III. IMPLICATIONS FOR THE KOLE OF NIE 

1. Procurement Behavior 

A. Orchestration 

B. System Building 

C. Process Mode of Management 

2. Considerations, Internal to NIE and Non~Procurement 
Activities I 

A. Staffing ( 

B. Internal k^ncy Structure 

C. Non- Procurement Activities 

/ 
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RKStiARCIl 

Research can be viewed as being essentially of :wo types: 

1. Basic Research: Research to produce knowledge for its 
own sake* 

2. Problem- Focused Research: Research seeking to produce 
knowledge applicable to the soi"tion of a specified 
problem (commonly known as Appixed Research) . 

In this section^ we will examine Basic Research. In the next section, 
we will examine Problem-Focused Research* 

BASIC RLSEARGI: KN(^LEDGE FOR ITS OV/N SAKE 

I > GENERIC CHARACTERISTICS OF BASIC RESEARCH 

The characteristics of Basic Research have been well and frefluontly 
discussed in the literature. !7e will here attempt only to highlight 
. briefly and call attention to some of the particular characteristics 
that arc of some importance to this analysis and which bear repeating 
by way of introduction. 

1. Uncertainty and Unpredictabi xity 

The primary characteristic of Basic Research is its uncertainty 
and unpredictability. Vox example: 

1. is very difficult to predict the form, the typt , the 
timing of outputs--or even oT the input,; that Will ho 
required. 

2. In vhe long run, what arc laior seen as the 'Wst important" 
bonelits of Basic Research were not even initially visualized^ 
bat rather resulted from spin-offs of the Bas*r Rosea rch or 
from findings that arose in une>;ectod atul unplanncd-lor 
turns and byv^ays in the Uoscaroh process* 

3. Indeed. t\w very definition or spc ification of what constitutes 



a ' Field of Research" tends ta change over time. 
A discussion of the uncertainty and unpredictability of Basic 
Research could be continued For some length. Wh;it is i.vjportant here 
is to rccogni::e some potential implications for Nil;. 

1. The Basic Research process calls for a high degree of 
creativity from Research personnel. The specific 
implications of this fact will become more obvious later 
in our discussions where the personnel involved are a 
significant consideration (e.g., in determining a 
"minimum critical mass" of talent). 

2. Long-range program planning is essentially meaningless 
on a project -by-project basis. Most particularly, it 
seems difficult if not l^pssible to program the kinds 
of sudden, major "breaHthroughs" that are sometimes 
sought. Indeed, in some fields or in some projects, 
ituch 'breakthroughs" may never occur. Rather, progress 
may be incremental, developing through an accumulation 
and extension of knowledge over extended periods of 

t irae . 

3. Since it is relatively difficult to know where and when 
useful outputs will occur, the primary need in Basic 
Research is for as much high quality activity as possible. 

4. It is important to be sensitive to what is happening in 
a field and how a field is changing, as contrasted to 
thinking in terms of programming the changes in a field. 

2 . 1*2 Fr ame f or Outputs 

Basic Research has l)Oth a short -tcnn and a loiH',-torm Frain.^. I 
tlio bhort term, the result of Basic Kcs'oarch is osr.c.itially to in- 
crease the tcnal_basc oj jvno^^ i.e., to product' kno.Jedgo that 
is in turn used and built upon hy other Basic Rt'searchtrs . U is 
^crur.'iUy only over very lonj'. periods of time (e.g., SO yv.irs) tiiat 



the Basic Research results in practical application . Thus, what is 
done in the present, is done for the future. Wh.it i lacking in the 
present is a result of omissions of the past. 

3» The Rate at Which a Basic Research Area can be Bullet Up 

Two points must be made about the process of "building up" a 
high quality Basic Research area* First, a high quality Basic 
Research area cannot be built up very quickly. The cumulative 
development of a total base of knowledge is generally a long-term 
process (as we have just noted). And it takes a considerable amount 
of time to both train competent Researchers and develop a community 
of Researchers who are committed to a particular field* 

Secondly, the rate at which a Basic Research area can be 
developed is dependent upon the number of competent Researchers and 
centers of excellence already existing in the field. If there are 
only a few corrpetent Researchers and centers of excellence, the 
rate at which the total base of knowledge can be expanded is rather 
significantly limited. The training of new personnel who arc 
committed to the field must be done by the currently available leading 
Researchers within the context of existing centers of excellence. Thus, 
the fewer such centers of excellence, the slower will be the rate of 
increase in the total number of competent, committed Researchers. 

The above considerations have some significant implitatior for 
the funding in a Basic Research area, llrtien the existing quality 
Research base is small, pumping large amounts of money into Basic 
Research would tend to have very little constructive or meaningful 
impact in either the short term or the long term, instead, 
the impact would tend to be the generation of a Iprge amount of 
lesser quality activity, much of which would simply he unproductive 
and would disappear (for lack of commitmcnit) when the funding uat; 
withdrawn* 

Beyond the setting of some upper and lowor l)ouiKlnri os, tht're 



r 



are virtually no useful guidelines for determining the ^^optimal" level 
of funding tor Basic Research « Rather > funding becomes essentially 
a matter of faith related to the availability of funding* It is not 
even very helpful to make analogies to^ other fields because: ^ 

!• These other examples were tHomselves based on this kind 
of faith and/oi;^ historical extension* 

2. The existing situation and opportunities can vary so 
dramatically as to make projections very difficult. 

An upper boundary for funding would be determined by the rate 
at which funds can be usefully absorbed by the existing centers of ^ 

excellence — i.e*, the rate at which they could increase the ftumMii: ^ 

of graduate students and/or take on additional Basic Research programs. 
The lower boundary for funding might be determined by the miniioal 
funding necessary to protect the existing centers of excclience (or, 
if necessary, to create such centers)* In setting both the upper 
and lower boundaries, consideration should also be given to funding 
bein^ provided by other funding sources (e.g., other federal agencies, 
foundations) • 

Between the upper and lower boundaries, tlicre would tend to be 
a rather wide range of levels at which funding could potentially have 
significant impact. It should also be noted, of course, that these 
upper and lower boundaries might (for practical purposes) be further 
constrained Jj^y economic, political or other non-Research considerations. 

4. Some bther Fundamental Issues 

A. The Criterion <of Excellence 

The criterion of Excellence is central and predominant as a basis 
for decision making. Ideally, at least. Excellence is the key selection 
criterion for programs and projects to be supported, funded, or oven 
permitted to take place (e.g., by allowing access to scarce facilities, 
etc.), and for the support of institutions and personnel. 
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B. Competing Centers of Excellence ^ 

However, where a Basic-Research field is relatively immature— 
where it lacks cohesion and consensus on the dcfijution of Research 
areas, clarity on the standards that define Excellence, a well 
developed social structure that establishes a picture of who the 
field regards as authoritative, etc. --the choice among those centers 
of excellence that do exist becomes that much more clifficult. Under 
these circumstances, funding agencies are faced with a choice of: 

a. spreading resources around to many centers of excllence; or 

b. "placing their bets" on a limited number of such centers. 

C. "Minimum Critical htass" 

The resolution of choice among competing centers of excellence 
requires (at least) the introduction of another premise concerning 
Basic Research--namely the need to maintain a "minimum critical mass*'* 
of effort. This premise holds that because of the uncertainty and 
unpredictability in Basic Research, because of the need for a creative 
interplay between a number of persons with different perspectives 
(sometimes disciplines, etc.), and because Basic Research is a 
"building upon," generally incremental process, there must often be 
a certain "critical minimum" number of Researchers interacting with ' 
each other if Basic Research is to be productive within a given time 
frame. This minimum critical mass may vary across fields and is most 
particularly applicable to empirically based work. Observations of 
prior successful patterns can permit some cstima«.ions to bo made. 

Taking together consideration of limited fimJing, the existence 
of several competing centers of excellence ami some notions of minimum 
critical mass, a funding agency mip,ht consider a strategy of "placing 
bets" on a limited number of centers of excellence as being superior 
to spreading its limited funds around to many renters of excellence. 

^' • A Pqs sibl e S et o f Se I cc^t ioii_ t rj ter i a 

Su-ama riling fronKa number of points in tlie uiiuiission thus far. 
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we suggest the following as a possible set of criteria for selecting 
Research areas for funding: 

1. Which disciplines (or sub-'*Uciplines) have a potential 

for making a contribution in the long run? 

IVhich areas are already being well -supported by the 

Agency or by other funding sources? 

3. Which areas have the sufficient seeds of excellence to 
build upon? 

After having made a determination of these first three criteria, then 
a fourth criterion may be added: 

4. Which of the areas have the best potential for building 
long term capability for the Basic Research function? 

BASIC RESEARQI IN EDUCATION 

1. -Veakness of Central Core of Educational Researchers 

The central core of Basic Researchers, those conunitted to and 
devoting the bulk of their careers to educational Research, seems to 
be particularly weak in this field, especially when compared to 
Basic Research in most other fields. In education, this core group 
tends for the most part to be located in schools, of education, in 
departments focused on each of a number of derivative disciplines 
(e.g., educational psychology, educational sociology). Consideration 
of these settings as the primary institutional bases for Basic 
Research in education suggests that there are relatively few centers 
of excellence and a great deal of mediocrity. Also, when one makes 
the distinction between numbers staffing these departments and number 
carrying out significant amounts of Basic Research, the relatively 
small size of this core, and its scattered condition, become apparent 

A somewhat different picture emerges when one examines Basic 
Research relevant to education carried out in discipline-based uni- 
versity departments (e.g., departments of Psychology or Soc.ioloRy). 
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Here, there is considerable Excellence, and these disciplines often 
provide valuable inputs and contributions to the education knowledge 
and technology base. The problem, however, is that the ccnunitraent 
of discipline-based Researchers to the field of education tends to 
be variable and shifting over time. After ^11, their primary com- , 
mitment is to their 'discipline, and education takes a secondary role. 

Taken together, these two conditions create an educational 
Research community, that . tends to be, unstable and amorj'hous, thus 
complicating the problem of relationships and of maintaining 
communication flows among the parts .of the system. What this implies 
for NIB is a major system-building need in order to (a) crea^- the 
vital, stable core of high quality Researchers and (b) facilitate 
and sustain interdisciplinary communication and collaboration. 

2, General Climate . 

The general climate surrounding Basic Research in education has 
not been supportive. Basic Research in the social sciences is not 
held in high repute by the public or its representatives in Congress. 
In education the problem is intensified by the difficulty even 
Researchers have in pointing to more than a handful of significant 
developments that are traceable to Basic Research in the field. The 

1 

consequence has been low prestige--complicating efforts to attract 
strong Research talent to the fiold--and low political support, ^ 
unreliable funding, and lack of continuity in funding emphases-- " 
complicating the problem of sustaining the work of those strong 
Researchers who have been attracted to the field. Such conditions, 
then, have tended to limit the quality of relationships between NIE 
and the Researchers, and have act'^d as a constraint on building the 
needed central core of educational Re:- earchers . 

5 • The In tcrdisc ipl inarity of Hd uc a t ion 

I-ducational Research tends to roquiro ;.n intordiscipl inary attack. 
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yet much of educational Research tends tg be discipline-focused. 
This issue adds to the difficulty of developing and finding con- 
sensus and hampers communication between the sub-groups in the field. 
Further, the conununication mechanisms in educational Basic Research 
are very diffuse. Lacking for the most part ore eithef the in- 
visible colleges or core journal mechanisms that typically structure 
information flow i" other Basic Research Cield and would simplify 
information searches across discipline lines. The relevant journals, 
both those from the disciplines and those within education, are so 
numerous and so inadequately abstracted that the cumulative develop- 
ment of a relevant interdisciplinary knowledge and technology base 
seems difficult to envision unless a lead agency such as NIE inter- 
venes and takes some facilitative actions. 

4> The Funding Sources for Education Research 

Given the multi-disciplinary nature of educational Research, 
and the ^nther substantial numb^jr of relevant disciplines, the single 
most prominent policy-relevant feature of the funding of Basic Research 
. in education would seem to be the existence and use of multiple funding 
sources. Looked at in total, NIE's potential contribution in dollar 
terms is relatively small. Tliis fact puts a premium on the require- 
ment for NIE to be aware of and stay in close touch with the various 
sources that do provide funding for areas of work relevant to education. 
In this way, NIE can effectively use its resources through strategies 
of gap-filling, attempts to pool and coordinate resources across 
agencies, etc. And given NIE»s position as the lead agency for 
educational Research and Development, this leadership/ cx)ordination 
role seems central to its mission. % 

IT 1 . 1 MPLICATIONS FOR THE RO LE 01- NIE 

^' Procu rement Behavior 

Fnci iitative Orchcst ralion 

Given the conditions wo have boon considering, the role of 
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NIE would clearly seem to be to orchestrate the various elements ^ 
in the system. However, in contrast to the kind of orchestration 
we will describe later ior the Development function, in Basic 
Research the kind of orchestration that would seem most appro- 
priate would need to take place through a combination of: 

1. facilitating the quality activities that are already 
existing ixi, the field (i.e. seek out and support 
Excellence); . ' 

' 2. selecting the Research areas with which to work; and 
3. being responsive to the shifts and changes, that are 
developing in the field. \ 

^uch a strategy of fdcilitative orchestration would require 
a close NIE relationship with the field—ongoing, one-to-one 
relationships between Agency staff and Bf»sic Researchers in 
the field. This kind of orchestration cannot really be. 
achieved through the use of advisory panels representing the 
diverse perspectives of the Basic Research community, even 
the more creative use of long-term panels for shaping and 
monitoring broad programs. Facilitative orchestration re- 
quires certain kinds of internal staffing in the Agency and 
a collaborative^'mode of Research liianagement that this kind of 
staffing makes possible. We shall return to these points 
shortly. 

B. System Building 

We have emphasized throughout our analysis the need for NIE 
to play a system-building role in managing its Basic Research 
program. We have suggested at several points that a key criter- 
ion in considering contemplated Agency actions with regard to 
this function should be estimating likely consequences for the 
development of long-term Basic Research capability— a building 
process that requires lengthy time spans and is constrained 
by the scale and state of development of existing centers of 
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excellence. If NIB accepts this role, then in project selection 
this might mean accepting the legitimacy of latent system- 
building purposes in procurement, and recognizing that the sys- 
ten-building consequences of a given project are likely to be 
of greater importance than the project itself or its anticipated 
subst^tive output. Once an agency accepts a system-building 
role, and is operating in a function such as Basic Research where 
system-building activities cannot be planned or carried out in- 

« 

dependent of the field and Agency actions can be of only the 
facilitative/collaborative type if they are to be productive, 
then the Field~ Initiated vs. NIE-Directed issue loses its meaning 
and forces consideration of a broader and more complex array of 
options. •* 

C. Implications oi Levels of Consensus in ^ j^ield 

Different Basic Research areas are likely to vary in levels 
of coiirensus among Researchers in the field— as to key questions 
in need of answers, adequacy or appropriateness of different 
methodologies, etc. Under conditions of low. consensus, the role 
of a funding agency would seem to be to work with the field 
rather than be directed by it— be fairly active in molding and 
selecting from what the field has to offer rather than just re- 
sponding to scattered field-initiated proposals. On the other 
hand, under conditions of high consensus, such activity on the 
part of the Agency would seem to be less necessary. Thus, the 
Agency could operate in a mode in which one was responsive to 
field initiations. This perspective is somewhat different from 
that piesented by the Duffy et^ al^. 1976 memos, it is also worth 
noting in compaiison to the Duffy et al. memos that under condi- 
tions of high consensus there would not seem to be the need for 
RFPs that seemed to be suggested there. 

D» Process Mode of Management 

Up to this point, we have focused our attention on the early 
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Stages of Agency nanagement of Basic Research, i.e., the selection 
> of Research areaa, institutions, and projects to fund. We have 
not as yet considered how an agency monitors and manages com- 
ponents of its Basic Research programs after these selection 
decisions have been made. A few comments would seem to be in 
order here. ' 

Given the unpredictability and uncertainty inherent in' the 
Research .process, tight monitoring of Basic Research programs in 
accord with predetermined output specifications is clearly in- 
feasible. A more reasonable alternative than ^ight management 
of the Research product would seem ,to be monitoring of the 
Research process . However, given the nature of Research and 
especially of Basic Researchers, bureaucratic controls of the 
Research process would hardly seem to be in order. Instead, the 
problem resolves itself if the Agency has an internal Basic 
Research staff that is itself involved in Research and is func- 
tioning as an integral part of the field. Under such circumstances 
NIE become part of the quality control mechanisms operative in 
the field, and may be acting both to insure the needed degree of 
quality control on given pieces of work while also stimulating 
the field's develoinnent of self- control ling mechanisms that make 
an Agency role in quality control less and less relevant over time 
as the system matures. 

2. Considerations Internal to NIE and Non- Procurement Activities 

A. NIE Staffing 

The procurement behaviors we have been considering — facili- 
tative orchestration, system building, and process management — 
all require the kind of close NIE relationship with the field 
that suggests a rather particular type of NIE staffing. Speci- 
fically what is implied is staffing an Institute Basic Research 
unit with personnel who possess considerable substantive com- 
petence in the Research areas with which they might be working. 



as 



It suggests too that these NIE Research personnel might them- 
selves be involved in the Research process. This would seem 
to be vital if such personnel were to l>e sensitive to the shifts 
and revolutions taking place in the fiaW aiid'xo be able to 
discriminate substanj^i^fc advances and quality as compared to ab- 
ticrent"* ShiiPts and poor quality. 

Given those circumstances, it is quite likely that the skills, 
experience and interests of NIE personnel would strongly in- 
fluence NIB decisions about which Research areas NIE would fund 
or support. While this might be of some concern, it need not 
be considered a major problem. Given the large number of poten- 
tially relevant Research areas, and t^he virtual impossibility 
of determining which of these are most likely to be productive 
for the future, there is no »»right" answer to which Basic Research 
areas an agency such as NIE should support. There is no reason 
why an agency should not be opportunistic in selecting areas to 
support— taking advantage of the Capabilities of its personnel, 
and permitting them to work with the field in the Research areas: 
(a) in which they find the most that interests 'and excites them, 
the most of what they view as productive work going on; and (b) 
those Research areas they are most strongly qualified to work 
with (and in) themselves. 

What we are suggesting here is the need for NIE's Basic 
Research programs to be staffed by personnel who are themselves 
highly competent Researchers in any of a number of Basic Research 
areas of relevance to education. And further, we are proposing 
reviving the idea initially included in NIE's structure and 
operations in the form of its Basic Studies unit— the notion that 
NIE should support an in-house Basic, Research capability- -not so 
much because of the Research they will carry out per se, but 
because of the manner in which this could permit NIE to have a 
totally different relationship to the field. 

NIE Basic Research personnel, under these conditions, would 
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likely have an intimate knowledge of a number of relevant Research 
areas and the Researchers who carry out the work in those areas-- 
who they are, what they are doing, what they have done in the 
past, what approaches they use, what parts of the relevant know- 
ledge/technology base of the field they draw on. They would 
know how to use the informal communication mechanisms of a Re- 
search area, or where and how NIB should start in any conten9>lated 
efforts to facilitate the development of invisible college mech- 
anisms or other informal communication channels. And equally 
important, they would have a very clear sense of whatever standards 
exist for judging Excellence, and too, have a sense of the sorts 
of questions to be considered in judging Excellence or in putting 
panels together to guide such judgments. 

Perhaps most critical of all, with such personnel on NIE*s 
staff, th^ NIE vs. Field issue would lose much of its meaning. 
Rather than NIE and the field confronting one another as discrete 
entities with often different intexests, the boundary lines be- 
tween the two would be more difficult to define and NIE could, 
more adequately function as not simply an agency responsive to 
th« field but as (in fact) an .integral part of the field. 

How can an agency such as NIE attract the kind of Basic 
Research talent we have been suggesting is needed? The problem 
is by no means a sinqple one. On the one hand, the Agency wants 
to attract strong Research talent. But it is unlikely that such 
personnel would be willing to leave Research settings for Agency 
positions for more than short stints (unless, of course, NIE 
could provide an attractive Research environment that could both 
attract and hold such talent--a strong argument in favor of in- 
house Basic Research units) . On the other hand, long-term com- 
mitments to the Agency would seem essential if the familiarity 
built up in working with Researchers in a given area is to have 
long-range utility for the Agency. A certain amount of inter- 
institutional mobility may be inevitable, and given that, perhaps 



could even be capitalized on> e»g», by developing exchange 
programs for two-way flows of personnel between the Agency and 
, the field, with exchanges entailing long-term roles with in* 
herent continuity regardless of the institutional affiliation 
of an individual at a particular point in time. Joint appoint- 
ments aad collaborative Research are other possibilities. We 
offer these as merely illustrative options. Our basic point is 
that such seemingly internal Agency matters as staffing have 
significant implications across the whole range of Agency actions, 
&.d .may severely constrain Agency options* 

One point of caution should be underscored about the kinds 
of personnel NIE should and should not have carrying out these 
activities. We would caution strongly against the kind of very 
independent "stars" of the field who are likely to distort the 
field to their own image of where the field should be rather than 
facilitating and working with the field. This does not mean that 
there are not some kinds of "stars" who are also good facilitators 
and collaborators. Rather, we are emphasizing the danger of 
selecting the kind of "star" who would tend to pull the field 
only into his or her own image, and would thus tend not to be 
sensitive to and supportive of other areas of significant activity. 
It is important to recognize that there ^s a considerable differ- 
ence between the role (and hence the kind of personnel) needed in 
a Research institute that is concerned primarily with conducting 
Basic Research and the rather different role needed in a coordina- 
ting funding agency that is promoting Basic Research, as is the 
case for NIE. 

B. Non- Procurement Activities 

If NIE had the kind of staffing we hav- been considering, the 
Institute could carry out a number of critical non-procurement 
activities vital to its leadership role in an immature system. 

At the very least we would expect such NIE personnel to 
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carry out such activities as: conducting Research of their own, 
perhaps in collaboration, with Researchers in the field; develop- 
ing ver>* clear notions of what quality work is going on, and 
how it could be ;facilitated; spending a fair amount of time 
travelling to stay in touch with people in the field, and regain- 
ing in touch informally by telephone, etc.; attending or even 
sponsoring conferences and seminars; working with professional 
associations; and perhaps even facilitating the development of. in- 
visible colleges and various other communication and quality 
control mechanisms so essential to the cumulative development 
of high quality knowledge/technology bases. 

Beyond this, NIB personnel might well carry out such critical 
tasks as: mapping Research areas; providing critical syntheses of 
the state of the relevant knowledge and technology bases; re- 
viewing what work is currently going on where, what, approaches are 
being used, how these relate to state-of-the-art needs and what 
kinds of work still need to be done, what opportunities exist for 
what kinds of talent, etc.; etc. The preparation of such annual 
review documents would necessitate close NIB staff interaction 
and communication with the field, and could easily be the wedge 
that would enable the field to see NIB as an integral part of 
its functioning and a critical facilitator of i^ts development. 
But only, of course, if these reviews suggest^ thinking reason- 
al)ly congruent with the best thinking in the field— again sug- 
gesting the need for close collaboration in the development of 
such reviews. 

Such reviews can serve a numbe of purposes, not the least 
important of which mif.ht be use as a mechanism for attracting 
strong Research talent in certain areas. If such reviews were 
distributed widely, and were sent especially to a carefully 
selected list of capable Researcheis whom one might want to attract 
1.0 work in the field of education, the recruitment and capacity - 
building potential of such mechanisms might be considerable. 
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This suggests a non -procurement behavior the Agency might con- 
sider, with considerable long-range implications for procurements 
as well as other aspects of, Agency action. This i by no means 
an original idea— NSF has been doing this with s ^ss for 

years. 

Mention of NSF points to one additional kind of non-procure- 
ment activity NIB should be engaged in— i.e., working t? bring 
about some degpree of coovdination among the various governmental 
and private ie,g», foundations) sources of funding for Basic 
Research relevant to* education. NIE might use its position 
as lead agency for Research and Development in education: a) to 
develop communication channels among funding sources; b) to 
develop analyses of the cross-agency funding pattern (e.g., 
where there are areas of overlap, where there may be gaps) ; 

« 

and c) to suggest opportunities for pooling of resources where 
the potential* for synergy among projects may exist and/or where 
such pooling of resources would seem to increase the likelihood . 
of greater overall' impact. 

I 

C. Relationship between NIB Staffing, Non -Procurement Activities, 
and a Wider Range of Procurement Options ^ 

To illustrate how the kind of staffing we have been consider- 
ing and the types of non-procurement activities this staffing 
makes possible, can impact on Agency procurement options, we 
borrow another example from NSF. If staffed with personnel 
having the proper credentials, we could envision NIE pursuing 
an interactive relationship with the field. An Agency staff 
member might carry out a number of intensive discussions over a 
certain period of time with some of the strong Research talent 
that exists within the field or in other relevant areas. These 
discussions might focus on the kinds of work seen as needed, 
jthe state of the art for carrying out such work, and what portion 
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of all this a particular Rec. .archer would want his own organiza- 

» 

tion to carry out. At some puir^t along the way, some of these 
Researchers might be approached to prepare brief statements-- 
the outlines of thinking oriented in the direction of formal 
proposals. (Or such initial statements might be derived from un- 
solicited proposals received by the Agency.) This brief state-% 
ment might be the beginning of a lengthier cycle of communication 
and interaction focused on collaborative development of proposals 
that reflected the thinking and interests of both the Agency and 
the Researcher (s). The Agency role here would be largely facili- 
tative. The NIB staffer might suggest that the Researcher^ get in 
touch with a particular organization that is doing some thinking 
along similar lines, or might suggest that the proposal could be 
strengthened or made more attractive for funding if this or that 
were changed in this or that manner. After several such cycles, 
with perhaps a certain amount of involvement of the broader field 
as well PS the particular Researcher and staff member, some rathet 
exciting and strong plans might evolve, rather different from what 
would instead have been proposed without this interactive, col- 
laborative relationship between NIB staff and the Research com- 
munity. We include this example simply to be illustrative of the 
range of options available for funding Basic Research, options 
rather differen;". in nature from dependence on the unsolicited 
proposal or grants competition mechanisms— with rather different 
implications for the manner in which NIB relates to the field. 

We recognize that the pattern of Agency/Field relationships 
we are offering here for NIE consideration runs counter to 
strongly held feelings in government circles about the need for 
"fairness"— the need to treat all potential contractors and grant 
recipients alike, without giving undue advantage to one or an- 
other individual or organization. However, we would argue that 
many of these convictions about "fairness" and "equal treatment" 
for potential contractors may be more appropriate to mature 
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systems, and inappropriate to Agency behavior in relation to 
imnature R/D^I systems. If NIB is to take seriously its legis- 
lative mandate to "build atj effective R§D system" in the edu- 
cation sector, system-building consi'^erations may have to be 
given precedence over other principles of Agency behavior— 
especially when those principles may be inappropriate for the 
particular context in which the Agency must function. We suggest 
that these matters need ca«eful consideration in relation to 
the rather fundamental kinds of questions we are raising about 
NIB»s mission and role vis I vis the fj.eld with which it works. 
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PROBLEM-POCUSED RESEARCH 

I, GEiNERIC CJURACTERTSTICS OF PROBLEM- FOCUSED RESEARCH 

1 . Introduction 

Problem-Focused Research has two basic characteristics — and 
herein lies an inherent problem. 

1. Problem- Focused Research i£ Research. 

2. But Problem-Focused Research is "targe tted*l— i.e., there is 
a very specific area of focus. Thus Problem- Focused 
Research seems like Development. 

. From the above, we can see that Problem-Focused Research can be 
deceptive— it looks both like Basic Research and like Development. 
It has elements of both; yet it is neither. Thus, the inherent 
tensions between Basic Research and Development are inherent in the 
Problea- Focused Research process. And this tends to lead predictably 
to certain kinds of problems. 

The basic tension is that: 

Ik On the one hand. Researchers tend to treat' Problem- Focused 
Research in a Basic Research mode. 

2. On the other hand. Users and Funders tend to treat Problem- 
Focused Research as if it were a Development activity. 

For the funding agency, some important issues arise from this 
basic tension. The tendency of Researchers to use a Basic Research 
mode even while conducting Problem- Focused Research may imply the need 
for some sort of monitoring role from the funding agency. But this 
would be a difficult role for an agency to '';irry out because of the 
unpredictability involved even in Problem- Focused Research. 

Perhaps even more importantly, the funding agency will have to 
resist its own temptation to treat Problem- Focused Research as though 
it were a Development process. Development permits a greater degree 
of specificity and so of Agency controlling activity. The point hero 
is that different- -not similar- -control mechanisms and strategies 
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are applicable because Development and Problem-Focused Research are • 
dissimilar. 

2. Users and Funders: Seeing Problem- Focused Research as a Develop 
aent Activity 

It is to' be expected that Users would tend to see Problem- 
Focused Research as a Development activity. After all, their concern 
is with the development of a product they can use . Perhaps to a 
lesser extent, we would also expect Funders to have a similar per- 
spective, especially given the fact that Researchers often oversell 
their efforts and the potential immediacy of the benefits as a 
strategy to obtain funding. 

Several problems tend to arise when Users and Funders think of 
Prob^em-rocuse4 Research as a Development activity. 

^1. In Development activities, there are specified "targets" 
, which a^e relatively fixed. However, simply becaus** 

Problem-Focused Research is Research, the "targets" tend 
to be unstable. What starts as, a Problem-Focused Research 
project may gravitate towards a Basic Research project 

(or vice versa). 

Because Users and Funders have a Development perspective 
(rfhich assumes a stable or fixed "target") , the shifting 
of "targets" in the Problem-Focused Research process easily 
leads to frustration by Users and Funders. 
2. Like any Research activity. Problem- Focused Research has a 
high degree of uncertainty and unpredictability in rela- 
tionship to time horizons, costs, utc. Again, this tends 
to be less so for Development activities. 

mien we consider the uncertainty and the instability of 
"targets," we can understand that the RFP-type of mechanism 
is really inappropriate for Problem-Focused Research. The 
RFP mechanism, with its specificity, creates an inevitable 
tension situation, forcing the Researchers to play games. 
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The Researchers will ten\J to oversell, to predict what they 
cannot predict in advaAce, ttf^^uggest they will produce " 
something about which they are uncertain. Once having been 
funded, the Researchers are likely to find they are pro- 
ducing something very different (or that they are working 
in a very .different direction) from what the RFP specified. 
Thus, the Researchers face a constant choice between the 
"lesser of two evils": either (a) decline the funding 
because they cannot' be sure they can meet the RFP specif i- 
cationsi; or (b) respond to the RFP and play games with the 
funding, agency. 

Problem-Focused Research is performe.' by Researchers who 
behave lilce Researchers rather than like Development 
personnel. For example, while Development personnel con- 
tinually have to face such requirements as specified time- 
lines. Researchers are not accustomed to being subjected 
to such requirements, and will thus tend to ignore and/or 
resent these "Developi ^' l type" requiVements 
In the Need Identifical /Problem Definition process, 
there is a real gulf bucween the Research and Development 
perspective*— and thus between the perspectives of Researchers, 
on the one hand, and Users and Funders, on the other. 

Users and Funders will tend focus on the practical 
problem immediately at hand. iTieir concern wi.U be: 
"What is- going to solve this problem?" Contrarily, the 
User/Funder concern may or may not be seen as important by 
the Researcher. The Researcher will be concerned with 
identifying a resea^chable problem- -which may or may not 
be the same as the problem identified by Users or Funders. 

Even when the Researcher and the User/Funder are, in 
agreement as to. the desirability of a specific problem- 
focused target, the Researcher may see only one aspect 
,of that problem as being r e s ear chab I e — and the Users and 
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Funde^fS-inay respond: "Yes, but that won't help us 
to solve our problem"; or, "Yes, but that would take 
too long." • 

The issue for the funding agency is the need to balance, on the 
oi^e hand, the significance of the problem area as perceived by Users, 
and" on the other hand, the extent to which that problem area is 
researchable, given the current state of the art. Hie tendency will 
generally be to put too much weight on the significance of the proolem 
area and the practical (end-result) justification for beginning a 
Research program, and to give inadequate attention to the extent to 
which the Researcher could likely come up with anything useful at 
this tine, or researched in this way, or researched in a certain 
time period. Probably the worst of all situations is to have very 

* 

bad Research being done on a very important problem. 

The nature of Problem-Focused Research is such that the User 
does have a significant role in defining the overall context for the 
Research— i.e., in identifying and defining the practical (User) 
need on which the Research is to be focused. However, the User's 
r^le must be kept in perspective. The User cannot determine what 
is in fact researchable, within what time frame, etc. Thus, the 
funding agency has the role of: 

1. orchestrating the tension between. User and Researcher 
perspectives; and 

2, rot^ undertaking program Research in areas that (for 
whatever reasons) cannot be usefully researched. 

3. Problems from the Perspective of the Researcher 

Switching now to the perspective of the Researcher, we find 
that several more problems may be likely to arise because of the 
Researcher's tendency toward a Basic Research rather than a Problem- 
Focused Research mode. 

* . . 1. Problem- Focused Research tends to require a scale and 
cost of efforts which is significantly greater than that of the 
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Basic Research mo4$ to which the Researcher is likely to be accustomed. 
This larger scale of effort tends to result from the fact that in 
comparison to Basic Research* Problem-Focused Research: 

« 

a) usually requires significantly more empirical investigation . 

» 

(either laboratory or field study); 

b) tends to require more usage of large and possibly inter- 
disciplinary teams. 

What rather naturally tends to happen is that the Researcher 
subsumes or redefines Problem-Focused Research issues into Ba^ic 
Research issues which can be handled within a smaller scale of efforts. 

2. Similarly* the Problem-Focused Research project typically 
needs to be done in sowd sort of large scale (somewhat bureaucratic) 
institutional complex (such as a Research institution, a Research 
laboratory) where larger scale, longer-term efforts can be made and 
continuj^ty maintained. Again, this is mode which is not typical 
of Basic Research. 

3. For several reasons, quality control tends to be far more 
of a problem in Problem- Focused Research than in Basic Research. 

a) For one thing, there are fewer and less effective control 
mechanisms. While we can find similar controls in relation 
to Problem- Focused Research, here the controls tend to be 
far more limited in niimber^ scope^ visibility and effective-- 
ness. Since self Iquality control of the field is weaker » 
the funding agency will here have to initiate and control 
the use of panels of people in the Research field* A major 
agency concern will be getting the right people together for 
these panels. IVhere there is a relatively weak Research 
community, the same persons will tend to be over-used ,/ thus 
creating problems of bias and narrowness of focus. Fik^ther, 
since Probl^.D<- Focused Research tends to be multi-^disciplinary, 
various groups can form around specific problems and self 
define the criteria of "quality*', in which case, relevant inter 
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disciplinary control mechanisms may not exist and may be 
needed . 

b) IVhen Problem- Focused Re5'»arch is interdisciplinary, each 
discipline will likely have different perspectives and 
conclusions as to piajWfem definition, criteria for "quality", 
and Research methodology. Thus, the very process of 
coalescing (and orchestrating) the different perspectives 
(and interests) into common agreement is itself a difficult 
process . 

c) The problem of quality control becomes even more significant 
when viewed from the perspective of scale and magnitude. 
Because Problem Focused Research is generally of a signifi- 
cantly greater scale or magnitude than Basic Research, 

the need for control in Problem-Focused Research is greater 
than in Basic Research— while the ability to control is 
less. 

At this pdint, the fUnding agency faces a very frustrating 
dilemma. The substantial funding investment required by 
Problem- Focused Research calls for increased control of the 
process, but because this is Research, the Agency canno.t be 
effective in trying to control the^ process bureaucratically* 
Thus, the primary means for an agency to control the process 
is to be part of the process— which in turn means having a 
significant internal Problem-Focused Research component. If 
an Agency had such a substantial Problem- Focused Research 
unit (or equivalent), these persons would be part of the 
self-regulating quality control mechanisms of the field. 
Instead of having bureaucrat ica I ly imposed control, these 
Agency Researchers would be working with the field in 
creating some standard field mechanisms, the kind of 
mechanisms that provide quality control in the disciplines. 
Because Agency Researchers would be part of the pr'ocess, 
the Agency would have a degree of control or involvcraent in 
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the field's own quality control, 
d) We may further note that because Of the greater scale and 
cost of Problem- Focused Research as compared to Basic 
Research, funding agencies tend to be more involved in 
the quality control process. Inevitably, this has the 
effect of Imposing more limits on the freedom of the 
Researcher. If however, this control were effected in the 
manner described above it might not be perceived as onerous. 

4. Tlie User*s Role 

Given the inherent tension between the Researcher and User 
perspectives in Problem-Focused Research, the need is apparent for 
some clarity in de:?ining the appropriate role for the User in the 
Problem- Focused Research process. This would seem to call for 
differentiating appropriate User roles in two stages of the Problem- 
Focused Research process; The Need Identification/Problem Definition/ 
Project Selection stage, and th^e Research stage. f 

1. In the Need Identification/Problem Definition/Project 
Selection stage, mechanisms are needed to allow significant User 
participation because: 

a) The User need is, by definition, a significant parameter 
in determining the focus or "target" of Problem- Focused 
Research. 

b) Tlie Researcher would normally have somewhat different 
insights and perspectives from those of the User. 

'^us, the pUirpose of User participation in this stage of the 
procc^.:. is to provide the Researcher with the inputs he needs to 
have a good initial awareness and understanding of User needs from 
the User's perspective . We should be clear, however, that hero the 
User role is not one of exercising veto power over project selection. 

2. In the Research stage, however, a rather different User 
role would seem to be called for--onc in which User participation is 



roininal. Here, the premium is on creativeness in response to the 
identified need, ai\d the Researcher is best left alone rather than 
influenced or controlled by possibly dysfunctional User pressures. 

S, Issues Related to the Characteristics of Personnel Involved 



We are now ready to delineate several key issues facing the 
funding agency as a result of cha|acteristics of the. Research*^ 
personnel involved in the Problem- Focused Research process. 



cai>ried out) and thus to be performed within an institutionalized 
bas^ the organizational context may be less than inviting for the 
Resear4iher. The extent of bureaucratic control of the Research 
process, and of such things as deadlines and programming of activities, 
is likely to be greater than that to which the Researcher is ac- 
customed. Having been socialized in Basic Research contexts and 
positions. Researchers tend to resist this kind of controlling 
activity. 

As a result, some of the better and more widely known and in- 
fluential Researchers will tend to avoid Problem-Focused Research 
in preference to Basic Research. 

Thus, the funding agency must recognize this dynamic and, if 
anything, bend over backwards to accommodate Researchers in order to 
keep them *«t^rac ted to the Problem-Focused field (in contrast to what 
agencies sometimes seem to do)., 

B. Redefinition of Studies into a Basic Research Mod e 

Even when Researchers do got involved in Problem -Focused 
Research, they will rather naturally tend to bend the Research to 
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the aodes with which they ^re comfortable, i.e., to redefine Problem- 
Focused Research projects into Basic Research projects. They will 
be more lively to look for the Basic Research aspect of the Problem- 
Focused Research ]problem than to look at the Problem^ocused Research 
problem ai; hand. 

Thus, the issue fdr the funding agency is to keep Researchers 
focused on Problem-Focused Research Concerns rather than following 
their natural tendency toward Basic Research questions. The funding 
agency must keep one point clearly and sharply in focus: Problem- 
Focused Research is Research, but it is not Basic Research . ^Thus, 
a Research proposal should not be funded as if it were Problem-Focused 
Research if it lacks the proper focus— i.e., if the proposal focuses 
on Basic Research questions rather than on the kind of grob^ran^. 
appropriate to Problem-Focused Research. It might of course be 
worthy of funding as Basic Research— from funds allocated to that. 
Because of the tendency of Researchers to switch the focus, the 
funding agency Kust be sure it is leading (in consort with the 
Researcher) rather than being led blindly. Its role is active and 
proactive instead of passive and reactive. Otherwise funds will tend . 
to be directed into Basic Research. 

This is precisely the problem so often encountered in field- 
initiated work (especially work suggested by unsolicited proposals). 
The initiators are often Researchers who, as we are noting, tend 
toward ^Basic Research, especially the most creative of them and those 
whose efforts tend to be focused on outputs capable of being published 
in the more prestigious Research journals. The potential dangers 
increase when the Researchers making such proposals are the "stars" 

*We are using "problem" here to refer to social problems or problems 
of practice, in contrast to the intellectual problems (i.e.. Research 
problems, or problems inherent in the development of the knowledge 
base of a discipline) that structure Basic Research inquiries. 
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of a field. Such persons are often Basic-Resoarch oriented, and 
being influential and prosti^jious, they tend to become field and 
f<$cus defining, and their proposals aro more likely to win approval. 
We must note here that this is not an issue of Excellence. The 
proposed research may be of excellent quality--it just may not be 
appropriately problejn-focused. 

y The funding agency must mediate the whole process, and thi's 
will require high quality personnel within the Agency. 

C. "^Problems in Maintaining the interdisciplinary Focus w 

Similarly, we must note that having been trained in a specific 
discipline, the Researcher will rather naturally tend to redefine 
problenls into those that fit into the perspectives and boundaries 
pf his own parti^lar discipline— the discipline in which he works 
most comfortably. 

It aay be necessary to allow this to happen at least to some 
extent, because this is what is of interest to (and thus motivates) 
the Researcher. The trouble is that it is too easy for the Researcher 
to shift the focus of a problem in this way. 

As a result, where a problem could be studied on an inter- 
disciplinary basis, the interdisciplinary focus may be lost unless 
some kind of proactive tension" is provided. 

Thus, the issue for the funding agency is to provide mechanisms 
to maintain an interdisciplinary focus wheres3|pssible, while at the 
same time being careful not to "ovOrprogram" such an interdisciplinary 
focus. The most reasonable strategy for the funding agency iix-JJus 
instance is probably to incJude this interdisciplinary requirement 
as part of the evaluation process at the beginning— during the 
selection of projects, the selection of institutions, etc. — but 
not to interfere with the overall process itself. 

The maintenance of an interdisciplinary focus is an objective 
most appropriately met during the selection stage; it is not likely 
to le accomplished effectively through attempting to impose process 
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controls. In order to maintain an interdisciplinary focus, the 
funding agency has two alternative approaches to consider: 

1. On the one hand, the Agency may choose to create an 
interdisciplinary setting in which the Research will 
be carried out. Here, the Agency must face the issue 
of whether* or not it wishes to get into an institution- 
building mode. ^ 

2. On the other hand, the Agency may choose to allow the 
various different disciplines each to undertake the 
work in its own way and within its own boundaries. Here, 
the Agency faces ^he problem of "putting it all together-- 
which may require additional projects, or the establish- 
ment of some other method of integration. 

V 

6 . Summary 

Inher<?nt in the Problem-Focused Research process arc two 
conflicting tendencies: 

1. The tendency of Users and Funders to turn it into a 
Development proce^. 

2. The tendency of Researchers tc turn it into a Basic 
Research process. 

A further complicating problem fes the fact that the Researchers are 
not a cohesive group but rather belong to different disciplines, each 
with its own perspectives and boundaries. 

The funding agency must be very clear as to the existence and 
nature of these conflicting forces. 

In this light, the funding agency has several key roles: 

1. to maintain the integrity^ the Problem- Focused Research; 

2. to mediate, balance, synthesize ths conflicting forces 
inherent in the Problem-Focused Research process; 

3. to insure that there is adequate input regarding the per- 
spectives and needs of the various kinds of Users to xvhich 



53 



/ 



the Research process is directed i 

4. to insure the participation of competent Researchers by 
presenting clearly and invitingly the Research aspects 
of the process which interest, excite and motivate them. 

It should be clear by now that balancing the conflicting forces 
in a way that keeps Problem-Focused Research productive requires 
skillful orchestration by the funding agency^ personnel. It is also 
immediately obvious that if the funding agency is to be able to 
perform its orchestrating role effectively, it would seem advisable 
for the Agency to haVe very knowledgeable aud skillful personnel on 
its staff— personnel who understand the forces ami issues involved. 
As this is not always possible, an alternative that might be attempted 
would be to utilize such integrating mechanisms as panels, conferences, 
etc., where representatives of the Users and of the various disciplines 
of the Researchers can discuss the issues and reach consensus. How- 
ever, we niust recognize that the basic nature of the Problem-Focused 
Research process is for the different participants to be in tension 
with one another. Thus, just bringing them together does not solve 
the problem of ccnflict— it merely provides a way or mechanism for 
orchestrating a conflict sitution. As a result, even when provision 
is made for bringing the different participants together, there 
remains^ the need for in-housc Agency people to perform skillfully 
the role of balancing, mediating, and orchestrati ig the conflicting 
forces . 

In the vvay the funding agency attempts to procure programs and 
manage Research activities, there is really no substitute for having 
within the Agency knowledgeable and. skilled personnel who are highly 
involved in the area of concern. 

The generic characteristics ^f.the Problem-Foousod Research 
process are so important that wc probably cannot overemphasize what 
Problem-Focuse. Research is and is not— and especially how it differs 
from 3asic Research and Development. 

* 
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Vrqm this perspective, we have ali'eady noted the following 

« 

implications for Agency strategy: 

1. The process^ cannot be controlled in a standard "bureaucratic" 
manner. Rather, an agency can best exercise control by 
being involved in the process. 

2. The Agency must play a major leadership role rather than 
being blindly led by the Researchers. At the same time, 
'Problem- Focused Research is not highly attractive to some 
of the most creative Researchers. Thus, while leading and 

« 

controlling, the Agency must also be as accommodating as 
possible to the Researchers in those aspects that do not 
subvert or divert the primary intent of th;6 program. 

3. • it follows that having knowledgeable professionals within 

the Agency is a very important part of a funding strategy. 

II. THE ErUCATIOSAL CONTEXT OF PROBLEM-FOCUSED RESEARCH 
1. Basic Weaknesses 

Probably the greater part of the Research that is carried on 
within the field of education is Research of the Problem- Focused type. 
There is a sizeable community of educational Researchers who carry 
out woik that is problem- focused. And there is an enormous quantity 
of Research of this applied or problem- focused variety produced every 
year. Nonetheless, the field is basically a weak field: 

1. Ntost of what is happening is small-scale, scattered, 
fragmented. 

2. There are major questions about the quality of what is 
being done. 

3. There is a lack of clear definition and identity of the 
field as somehow differen*. from either Basic Research or 
Evaluation or Policy Research. 

. 4. There is evpn a lack of clour delineation or consensus on 
the problem areas that structure this as a field of inquiry 
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or as an R/D§I function. 
There is a considerable amount of confusion in education as to 
just what is and is not Research of an Applied or Problem-rocused 
nature. On one end of the inquiry continuum,- one finds considerable 
difficult in education in distinguishing Applied or Problem- Focused 
Reearch from Basic Research. On the other end of the continuum, 
there is a tendency of many— generally non-Researcheri;— to confuse 
Applied or Problem- Focused Research with a broad range of other 
activities thi^t cannot properly be subsumed under the Research rubric— 
e.g., library research (such as examination of alternative sets of 
teaching materials); demonstration projects (a favorite device for 
diverting "Research" fUnds Into other uses); or the social bookkeeping 
kind of statistical record keeping that occupies the attention of so 
many school system "Research" offices. 

^ One of the clearest Indicators of the weaknesses and Immaturity 
of the Probie:3- Focused Research field is the widespread inability of 
educational Researchers to define what is a researchable problem. In 
e<»ucation, a great deal of what Is defined as a problem is defined on 
the basis of the significance of the topic area rather than on the 
basis of what Is researchable. Indeed, many of the proposals thit are 
made are about questions having clear significance in terms of the 
substance of the problem. However, the simple fact that a significant 
social or practice-based problem is not itself a Research problem is 
sv.n©thing that does not appear to* be completely understood in the 
field of education. The oft-noted^criticism of the inability of 
educational Researchers to define reasonable Research problems comes 
about in part because there is^ a gap between Researchers' undtorstanding 
of a given need or topic and how one would go about researching the 
need or even translating the general problem into a researchable 
problem. As a result, Problem-Focused Research in education is 
characterized by a tremendous amount of oversell. This is not because 
the reoplc involved are charlatans, deliberately promising lo answer 
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questions they kr.ow thoy cannot answer. Rather, it is too often the 

cai'.f that the Researchers do not even know that the state of the art ' 

renders thesj incapable of answering questions which they have defined • 
as a need or problem. 

2. Institutional Bases 

There are primarily two institutional bases where Problem- 
Focused Research is carried out in the field of education: the 
universities, and a number of large R^D organizations in the private 
and quasi-public sectors. There are problems inherent in both of 
these institutional bases . 

A. The Universities 

The universities have the capability to do Problem- Focused 
Research, but they tend to do it in a Basic Research wpde--both 
because tHat is the mode the Researchers have been trained and 
socialized in. and because of the social pressures of the university 
as an institution. In a university setting. Researchers are rewarded 
for acting like Basic Researchers rather than for problem solving. 
Thus, Research gets bent into the Basic Research mode, especially 
if the Research ia bcivig done by a group within a particular discipline. 
A second point about universities is that they usually do not have 

» 

a minimum critical mass of the capabilities needed to carry out 
Problem- Focused Research. The result tends to be that either (a) 
a problem that requires r large scale approach will tend to be scaled 
down to fit the resources, that titc university has available; or (b) 
as an alternative, the university will put together an ad-hoc group 
of people who tenu not to stay together very long as a work group. 
Neither of these alternatives is really very helpful, but both are 
typical enough to warrant some further attention. 

^ • Scaling Down the Prob lem 

If a university lacks adequate resources to dovil with a 

problem that requires a largo scale approach, the problem will 





tend to be redefined to a snuiller, manageable scale.. In • 
particular, the time frame will tcJ^d to be reduced (as we 
shall note in inore detail below). Additional ly, ' m a uni- 
versity setting a problem is often approached by a group 
of Researchers from a specific disciplinevand, rather natur- 
ally, this group will scale the problem dowin to fit into the 
focus and capabilities of their own disciplines, thus veering 
the Research away from the problem focus to a disciplinary 
focus. In schools of education, where the problem focus is 
more likely to be maintained, a somewhat different kind of / 
problem is encountered frequently. With the possible ex- • 
ccption of the ten to fifteen schools of education in this 
country with strong Research traditions, schools of education 
general! y^lack the kind of well trained Research faculty and 
graduate students more likely to be found within the- dis- 
cipUr.ir>' departments. Typically, then, the university setting 
.or Problem- Focused Research offers the funding agency a choice 
between well trained personnel who are likely to redefine Problem 
Focused Research into Basic Research modes or adherence to the 
problem focus by Researchers less likely to produce quality Work. 

.2. liio Ad-Hoc Approach 

Whether the reason be the disciplinary emphases of university 
personnel, the general fragmentation of the field, or whatever, 
it is the nature of the university social system that people 
will tend to come together for a while to work on a problem 
and then go their separate ways instead of staying together 
for a long period of time. 

This ad-hoc, short term approach tends to be highly dys- 
functional simply b'evause Problem- Focused Rcsc.irch (and 
Research in generall tends to require longevity--i .e. , the 
building of skill groups that can work together over long 
periods of time. The focus of the Research is problem solving. 




a process that takes time, especially if the need is to 
develop a long-range Problem -Focused Research program. 

Ke may also note that the commitment to a set or series 
of concerns (which tends to exist in .Basic Research in tho 
disciplinary setting) tends more often than not to be missing 
in relation to Problem-Focused Research--a dynamic that re- 
inforces the ad-hoc approach,. 

This tendency towards adrhoc approaches bears further 
examination from the instit.ution-building perspective. There 
may indeed be extr«nely well written proposals from a uni- 
versity ^roup to do Research in a given area. There may even 
be a •^star*', a very competent Researcher with the appearance 
of a str^ong group around him (or her) to do the Research. 
Indeed, for providing good quality results from a particular 
piece ^ Jlesearch, -this group may in fact appear to be a strongly 
qua Jki fled group, one that would do the work well. 



with long-term capacity and system building as well as with the 
particular results of a single Research project. Looked at 
from an overall system perspective, the procurement of a group 
that will be together only for one or maybe two projects will 
not be as relevant, meaningful or cost/effective (in the long 
term) as funding a group which has (or has the potential to 
develop) longevity. In the latter instance, the Agency would 
be procuring not qnly a short-term product but also a system- 
building capacity. 

In summary, universities tend not to be able to put together 
long-term, large scale work groups. As a result they tend cither 
to scale down a problem into one that is manageable with existing 
capabilities; or they develop ad-hoc, short-life groups for 
individual Research projects which add nothing of significance 
towards building overall system capabilities. 




would seem essential for NIE to be concerned 
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B. Urge R^D Organizations in the Private and Quasi -Pub lie 
. Sectors 

A somewhat different set of problems characterize Problem- 
Focused Research as this is carried out in large R8D organizations 
in the private sector (generally non-profit corporations and oc- 
casionally for-profit organizations) and in the quasi -public sector 
(the federally- created regional laboratories and R§D centers) . These 
are organizations that, by tj^eir very nature, should be particularly 
well-suited to carrying out Fx^oblem- Focused Research. They are 
likely to develop an^^ut together the resources for large scale 
Problem-Focused Research, particularly the larger organizations. They 
are likely to put together work teams that will stay together for a 
long time. -There are not the internal social pressure that one gets 
in the university sotting that would veer Res^rchers away from 
focusing on specified problems (even when these seem to provide no 
exciting and potentially publishablo outputs) . 

Howevpxvr at least two significant kinds of problems are dis- 
cernible from the history of these organization. First, it is not 
obvious that there are attractive career paths in these organizations 
that are capable of attracting the best talent. A small handful of 
these organizations win a large number of contracts, and can offer 
people some degree of stability and the prospect of a long-term career. 
But, compared to a university (which is the setting that tends to 
attract most of the best Research talent), there is no similar kind 
of long-terra tenure or commitment. The likelihood is that, for a 
variety of reasons, these organizations may have great difficulty 
attracting the strongest Research talents. 

The second kind of problem may be an even more serious one. 
Many of these organisations were 6riginally established to carry out 
Applied (i.e., Problem-Focused) Research as well as Development. 
However, the history of the labs and centers is suggestive of a 
pattern that is probably somewhat discernible in the private sector 
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as well. -By the later 60s, federal procurement emphases reshaped 
these organizations, especially the labs, into Development organiza- 
tions rather than organizations that carried out the full spectrum 
of R/D^I activities including Problem- Focused Research. That is 
to say, most of the money tended to go for Development rather than 
Problem-Focused Research. Consequently, these organizations (being 
the tj/pe that are structurally responsive to funding sources) ex- 
panded their Development capacity more and more. To the extent they 
carried on Problem-Focused Research, it might have been as an induce- 
ment to keep some of their Research talent happy and to maintain them 
in the organization. 

To summarize, there is one all-pervasive point that becomes 
apparent from examining the institutional bases of Problem-Focused 
Research in education: there is a fair amount of Problem-Focused 
^tesearch around, but it tends to be discrete, small scale Research 
projects rather than very large-scale projects. Certainly there may 
be a significant number of Problem- Focused Research problems which can 
be tackled in a small way. But the ones that are probably the focus, 
of NIE procurement tend to be larger-scale, complex programs tliat 
will likely require long-term Research capability. 

The basic point is that there Is very little large-scale, long- 
term Problem- Focused Research being done. The universities tend to 
scale it down or not do it in tKe long-term mode. The large insti- 
tutes and organizations in the private and quasi-public sectors that 
are organizationally capable of putting together large scale long-term 
work teams tend to have been pushed into becoming primarily Development 
organizations. This may be overstating the case; but certainly they 
have not filled the need for a significant amount of large-scale, 
long-torro Problem- Focused Research to meet needs. 

. It appears tliat Problem-Focused Research in education is 
essentially an area that needs to be put together- -it it not really 
there now to any sizeable degree. In part this is because the field 



has tended to think in terms of Basic Research concepts, or Develop- 
ment concepts, rather than thinking of Problem- Focused Research as 
a fundamentally different kind of Research, requiring different 
sorts of structures and instittitional bases which are suited to 
Problem-Focused Research as a distinct mode of RiD activity. 

3. Some Additional Corisiderations 
^ Communication 

Problem-Focused Research shares many of the problems that we 
already noted about Basic Research. The information mechanisms and 
the communication structures of the field are inadequate— even more 
so than in Basic Research. Not only does the field lack invisible 
colleges,but there is an even 'weaker journal structure. Relatively 
few journals specialize in definable Problem- Incused Research areas, 
and material relevant to any particular problem area tends to be 
scattered anong scores of journals. There are relatively few 
abstracting mechanisms in the educational field at all, and virtually 
no usable abstracting mechanism of relevance to any but a handful of 
problem areas. The whole information retrieval system, which might 
be potentially useful to the Problem-Focused Researcher, is really 
not geared to Problem- Focused Research. 

B. Commitment 

Given the way the field is currently organized, with the Problem- 
Focused Researchers constantly moving in an oiit of problem areas, 
commitments tend to be short-term rather than long-toim. Thus in- 
formal communication mechanisms and personal information flows are 
difficult to sustain; the communication networks are even more 
diffuse than in Basic Research, where we have already noted the vciy 
rudimentary state of development of comniunications networks. 

C, Climate 

The climate surrounding Problem- Focused Research is even more 
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negative than the climate surrounding Basic Research. The reason, 
quite simply, is that more tends to be expected of Problem-Focused 
Research, and therefore the disappointment is far greater when no 
payoff seems forthcoming. Basic Research does not hold out the 
promise of solving life's problems so saliently. Thus, there is a 
degree of acceptance when a specific Basic Research project does not 
produce answers— or may do so only in a distan future. 

But, in contrast, precisely because it is^ problem focused, 
Problem-Focused Research does (rightly or wrongly) seem to hold out 
a "promise" to solve ongoing operational problems. Thus, when such 
rather costly Research projects do not produce answers (or might 
possibly do so only over a long period of time), there i£ disappoint- 
ment. The result, inevitably, tends to be the generation of strongly 
held feelings that Problem- Focused Research is a waste of money. 

D» , Need Identification 

A related point here has to do with Need Identification for 
Research— who does it, and how a. whole' set of problem areas have 
been identified. In the past, Ne^d Identification has been 
Researcher-driven. That is to say. Researchers identified their own 
problems, with very little input from, Users. Thus, the problems 
defined by the Researchers were things that interested them rather 
than necessarily the problems perceived by Users or by people 
affected by the problem. As a 'result. Researcher-defined problems 
turned out to be either so irrelevant, ill-defined or misperceived 
that the User community perceived the Research as useless. 

Now it appears that there has been an overreaction to the pre- 
dominance of Researchc -initiated inquiry into problems. Thus, the 
definition of problems is now perhaps overly system-driven. NIE is 
defining problems; Users are defining problems; and the Research 
community is doing a gn it deal of complaining about the fact that 
the problems which are being defined either are not problems that are 
researchable or are not problems that should be the focus of Research 
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sponsorship. 

^ Clearly, then, there is some need for orchestration in the Need 

Identification process. Given the current level of maturity in 
educational R/DSI as a whole and educational Problem-Focused Research 
in particular, only an agency such as NIE can provide the orchestra- 
tion needed to maintain and mediate the existing tension, permitting 
the Researcher viewpoint to take on increasing influence as the 
capability of the Research community permits. 

E. Funding 

As in Basic Research, the sources ^or Problem-Focused Research 
funding are numerous. NIE, as the lead agency (even though not 
necessarily the major funding source), has a role in working with other 
agencies to orchestrate funding so that NIE funds can be applied in a 
way maximally useful to the system. 

I. Implic ati ons for the Role of NIE 
1. Procurement Behavior 
A. Orchestration 

Given the basic tension inherent in Problem-Focused Research 
and the resultant tendencies for Researchers to treat it as 
though it were Basic Research while Funders and Users tend to 
treat it like Development, NIE has a key orchestration role in 
mediating the inevitable conflicts between Researchers on the 
one hand and Users and Funders on the other hand- -as well as 
reconciling differences in perspectives among Researchers from 
the vapous disciplines relevant to a jgivcn problem area. ■ 

Critical here is the orchestration of User and Researcher 
perspectives in problem selection— balancing Users' interest in 
targetting Research at problems where they perceive a real need 
and Researchers' concerns about the rcsearchability of identified 
probicm areas given the state of the art . in relevant fields. 
If high quality Researchers arc to be attracted to and maintained 
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in large scale oducational Research institutions that carry out 
Problem-Focused Research, the field and the institutional settings 
nust be made attractive to them. As far as is feasible the forms 
and nodes of funding and control roust be made to accommodate their 
needs. It is vital to include the User perspective of problem 
area significance in initial project and program selection, but 
the overall balance will need to bo redressed in favor of the 
protection of the Researchers* criteria of Excellence and re- 
searchability (so* of ten lacking in the current situation). 

This kind of orchestration will require great sensitivity 
and ski 11 « and suggests the need for Agency personnel; a) who 
can work closely and well with both Researchers and Users; 
b) who can gain their trust and* confidence while still maintain- 
ing the integrity of the Institute's perspective; c) who can 
intervene appropriatol) to make certain that the Research 
Wintains its problem focus and that the problems selected for 
study both meet^User needs and are in fact researchable and 
carried out in a manner likely to be productive. 

The use of advisory panels to help NIB staff define Research 
program areas would seem appropriate. But their influence may well 
be somewhat limited until the field gains in strength. A danger 
to be avoided is the filling out of thcs*» panels with dispro- 
portionate representation from the Basic Research community (no 
matter how distinguished these Basic Researchers may be, especially 
those from the disciplines) and from the User community. Until 
more can be delegated to a stronger interdisciplinary field, NIE 
will have to be prepared to assume a larger share of the overall 
programming responsibility. 

B. System Building 

The fundamental point we emphasized in our analysis of the 
edjcational context was the relative absence of Problem -Focused 
Research settings with the required attributes of large scale. 
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longevity, and Excellence. Fur;ther, wo noted that prior funding 
policies had tended to transform the centers of applied work 
that do exist into what are primarily Development organizations. 

The implication for NIE is that institution building (and 
rebuilding) is a very important need--and only NIB (through its 
funding, orchestrating, and managing) can provide the impetus 
for such a strategy. 

Institution building implies: 

1. locating existing or potential centers of excellence; 

2. developing and s|upporting them in a manner that will 
permit the longevity of work teams and the kind of 
scaled operations that Problem- Focused Research often 
requires , 

Specifically in relation to universities (as one obvious 
source for centers of excellence), the problem is the tension 
between la) the need to keep the focus on Problem- Focused Research; 
and (b) the fact that Problem- Focused Research is generally not 
rewarded in the university setting. Thus, two alternative strate- 
gies could be: 

1. to attract competent university Researchers to R5D 
organizations in the private and quasi-public sectors 
(which, however, would have the dysfunctional effect 

of weakening potential university centers of excellence); 
or 

2. to provide joint-appointment arrangements wherein 
university Researchers> can both maintain their univer- 
sity relationships (and security) yet also devote major 
amounts of their time to Problem-Focascd Research in 
these R5D organizations outside the university setting. 

In relation to large-scale R(iD organizations in the private and 
quasi-public sectors, NIE's strategy will probably have to bo one 
of rc-building--i .e., developing funding policies with the needed 
focus, scale, and time- frame to permit these organizations to 
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return to the Problem -Focused mold. 

As in our discussion of institution building for the Basic 
Research function, system, building will require a somewhat close, 
collaborative relationship with the field to permit the identi- 
fication of potential centers of excellence with the required 
"minimum critical mass of talent." And as in Basic Research, 
with limited resources for funding and a wide array of potential 
problem areas for the focus of such funding, a policy of "placing 
bets" and concentrating funding on a few selected problem areas 
would seem preferable to scattering the available resources across 
a much larger range of problem areas and Research organizations. 

The funding of Problem-Focused Research in education has been 
characterized for some time by a Catch-22 problem i.e., a funding 
pattern which requires Researchers to promise unattainable (or at 
the ver>- least, highly uncertain) results in order to obtain 
funding. Thus, if you promise, you can't deliver. If you don't 
promise, you don*t get funded. 

NIE—and Users, Researchers, and the Congress— need to clearly 
understand that: 

1. The problems selected must be important problems. 

2. At the same time, the problems must be researchable. 

3. The capacity does not really exist right now to 
meet those needs. 

4. Thus, institution building (and rebuilding) is the 
prime need. 

5. However, institution building is an ongoing, long-term 
process. 

These points imply that Problem- Focused Research must be 
promoted as an important but long term contributor to education-- 
onc that requires a major investment now^tf the gap is to be 
filled, more for -future than for present benefits. Perhaps this is 
the only way to overcome the Catch-22 J^roblcm. Vital to such a 



strategy is that Users, the Congress and NIII have a clear under- 
standing of the differentiation between Problem- Focused Research 
and Development. 

C, Process Mode of Management 

Given the size of the investment required for large-scale 
Problem-Focused 'Research, and recognition of the somewhat limited 
statfe>jP development of the field, a funding agency is likely to 
hav^ a strongly felt need to exercise some control over its Re- 
search program. But, as we have emphasized repeatedly, Problem- 
Focused Research i8_ Research, ind it is carried out by Researchers. 
Therefore, it would seem essential for NIE ttnlvoid a bureau- 
cratic response. Tightly drawn RFP*8 that attempt to mandate 
targets, deadlines, procedures, etc. have limited likelihood 
of being effective and seem to be inappropriate substitutes for 
knowledgeable close interactive relationships between NIE staff 
and the field, especially where NIE staff include Researchers 
ivho function as an integral part of the field. 

2. Considerations Internal to NIE and Non-Procurement Activities 
A. Staffing 

Recd^izing the high cost of establishing large scale Problem- 
Focused efforts of appropriate critical mass, the visibility of 
such efforts, and the tremendous difficulty of keeping such work 
problem- focused (especially during a period in which the emphasis 
is to bo on institution building and on attracting strong Research 
talent), NIE will be faced with the need to build a very strong 
internal group capable of close working relationships with the 
field--rclationships that imply a substantial degree of leadership. 
This can be achieved only if such personnel are themselves involved 
in important Problem-Focused Research and have achlcve.1 some stature 
in the Research community. 

Without such a relationship nnd a strengthened field (backed 



68 



up by the needed in-house competence and activity), merely doubling 
or tripling the proportion of funding allocated to field-initiated 
Research would not seem to be a purposeful strategy in and of it- 
self. Unlesi NIE is capable of orchestrating the Basic Research/ 
Development 'tension, rhe quality control requirements, the 
institution building programs and the much greater amount of fund- 
ing available for Problera-Kocused Research from other agencies 
and, further, is capable of performing the monitoring of the total 
effort, then merely turning more of the initiative over to an 
overtly weak field would seem to be futile. 

B. Internal Agency Structure 

Given what we have said of the importance of separating 
Problem- Focused Research from both Basic Research and Development^ 
it might be desirable for this se'paration also to be reflected 
in both organizational and budgetary structures within NIE. This 
would seem to be a matter that merits some attention within the 
Agency . 

C. Non- Procurement Activities 

Much of what we discussed earlier in relation to non-procurement 
activities of the Agency appropriate for the Basic Research func- 
tion applies equally well to Problem- Focused Research. Included 
here mignt be such activities as: NIB Researchers conducting 
Problem- Foe used Research of their own, possibly in collaboration 
with Researchers in the field; keeping in touch with the field 
through travel, telephone communication, attending and/or sponsoring 
seminars and conferences, etc.; facilitating the development of 
invisible colleges and other communication and quality control 
mechanisms needed for the cumulative development of relevant 
knowledge and technology bases; etc. 

Here too, NIC Researchers might map potential problem areas, 
prepare syntheses and reviews of the state of availabh' knowledge. 



etc., and use such documents for recruiting and attracting top 
Research talent to specific Problem- Focused Research areas-- 
in much the same manner described earlier in our discussion of 
Basic Research. 

The procurement implications of these kinds of staffing and 
non-procurement activities parallel those considered earlier 
for Basic Research. Here too, close interactive relationships 
between Agency and field, and ^ollaborative development of 
Research proposals and programs, would seem possible if system- 
building considerations could be seen to take precedence over 
the rules of the game of competitive procurements. Inherent in 
this discussion may well be one of the most significant and 
controversial policy issues in need of NIE consideration— one 
fraught with enormous implications for the Institute as well 
as for the whole educational R/DSI system and its future develop 
ment. 
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DEVElQPMtiNT 

TtiE NATURE OF nnVELOPMENT 

1. Introduction 

A. User Focus 

Development involves a process of converting knowledge into 
User^ready products.* The emphasis or focus of Developmient is the User. 
The end-product must be something a User can use (with at most some 
minimal fitting or tailoring). Thus, we use the term •^UsW-ready" to 
emphasize that when a product is "developed," it is in a form that the 
User can potentially use--it is "ready" for use» 

It follows rather obviously that Development is very dependent 
upon and thus must be very responsive to User^ needs. Further, we must 
emphasize that Development must be related to current User needs--not 
to some "foreseen" long*-term future needs. 

Of course, there is always the possibility to foresee (or project) 
a time when Users will be ready for a product which they Jo aiot currently 
see as a need. However, this kind of situation tends to be much more 
theoretical than real. Normally, a Developer has neither the capability 
to make such "putting products on the shelf" projections accurately nor 
the luxury to do so. • 

B. Dependence upon the Development State of the Art 

Whether or not it is even possible to develop a specific product 
is, of course^ determined by what the state of the art permits. Whether 
or not a product shou 1 d be developed also depends on the state of the 



*As noted in an earlier chapter, for simplicity of usage we use the term 
••products" whenever we are referrini? to the outputs of the Dcve;opment 
process. However, th '^eader should keep in mind that the term '•products'* 
is meant here to .convc^ the full array of Development output s--orograms, 
processes, mod<Ms, strategies, approaches, etc. as well as the narrower 
range of outputs wo typic«jlly think of as ••products." 
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art. For example, there is no point in creating a product that is in 
fact ouT-of-date if the state of the art permits a superior product to 
be developed. 

Ue must note that we refer to the Development state of the art — 
not to the Research state of the art. As we will note later, there is 
in fact very little direct transfer from Research into Development. Tlie 
products of Research appear in long-term, indirect ways. By contrast. 
Development products generally come from a more certain, ejjtablished 
knowledge base i.e.. Development products tend to be based upon or 
are an extension of other kinds of Development products. In simple 
terras, an existing product >s modified, improved, etc.. or a well 
developed concept is turned into a usable product. 

C. Development and Production 

Development is the creation of a product in a form in which it 
can be used directly or duplicated for wider use. 

In the generic sense, the end of Development is the beginning of 
Production. Once a product is developed. Production begins and at some 
later time (usually soon after Production begins), the Producer dis- 
seminates (distributes, markets) the product to the User. 

In some fields (where the products arc essentially technologies, 
e.g. in education), this clear-cut process of Development/Production/ 
Dissemina*-ion/Utilization generally does not exist. In such cases, the 
actual Production may be carried out by the Users themselves. After 
Development is completed, the next step is usually simply informing 
the User how to reproduce the developed product. Thus, in education, 
a clear-cut, separate Production stage often does not exist. 

Du T he Steps in Development 

The Development process will depend to some degree on the nature 
of the product being developed. However, i«i one form or another the 
process will generally involve: Need Identification, Development justi- 
fication and feasibility.! dcsrgn, model or prototype building, testing, 
packaging (which may include creation of Implementation/Utilisation 

■ ♦ 
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instructions), pilot dissWiination, large-scale field testing, final 
modifications. 

2. The Centrality of System Linkages 
A. Types of Linkages 

In Developraent, three system linkages are of critical importance; 
i.e., the linkages between the Developer and: 

1. the User (f^r^^ed Identification) 

2. the Oevelopmeiit state of the art 

3. the Production stage. 

/ 

1. Need Iden ti fication Linkages with the User 

^ . 

Normally we tend to think of Need Identification as occurring 
at the beginning of the Development process— i.e., when the User 
(either directly or through some intermediary) works with the Developer 
to define the area of need. Typically, Need Identification is then 
considered to have been completed. 

This, however, is a very limited concept, one that works well 
only under conditions of overall systcan maturity and certainty--i .e. , 
when: 

1. the Users can clearly specify what they need; 

2. the Developer (and Producer) can know exactly what the 
Users mean; 

3. the Developer is then capable of producing the developed 
product and then saying with assuredness to the Users: 
"Here is what you asked for"; 

4. it is then obvious to the Users what to do with the product. 
As an example, an airplane manufacturer may well be able to specify so 
clearly the requirements for a needed airplane part that the part can 

be developed to specifications and then, in effect, simply "plugged in." 

In contrast, of what value is such an isolated, simplistic 
procc.-s of Need Identification under conditions of overall system 
imnaturity and uncertainty (such as in education), when the User 
cannot clearly tell the Developer what is needed, when the Developer 
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vouia not really be sure how to go about developing the product oven 
if the User were to give clear specifications, etc.? 

Under these conditions. Need Identification nmsjt be seen as 
a broai. ongoing, continuous process — a process that: 

1. enables the Developer throughout the various stagcs%f 
Development to seek and receive additional information 
and clarification from the User; 

2. not only involves successive stages of the Development 
process, but is continuous through Dissemination, 
Implementation, and Utilization, until the need has been 
refined sufficiently for a usable product to be developed. 

In a word, under conditions of overall system immaturity and un- 
certainty. Development is a continuous prcess of adjustment and 
modification— a tailoring of the product which does not end With the 
generic "Development" stage but continues through the User aspects 
of the R/DSI process. 

2. Linkages to the Development State of the Art 

W^»4le not as criti'<?al or as difficult as linkages between the 
Developer and the User, th^ linkage between the Developer and the 
Development state of the art^ is nonetheless important. The Developer 
does need to know what has hc^n done, can be modified or improved, 
can be applied to a particular project. 

The linkage is to the Development knowledge base. This does 
not mean that linkage to Problem-Focused Research cannot at times be 
helpful. (Indeed, some Problem- Focused Research quite often occurs 
concurrently with the Development proccss--i .e. , as the Developer 
runs into "problems.") This also means that the linkage to Basic 
Research is at best tangential and occasional. 

5. Linka ge s to the Production Stag e 

A not uncommon breakdown in the R/Dtil process occurs when a 
dcvel^t)'>d product is given to the Producer in a form that (for whatever 
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reason) is nor practical or feasible for Production purposes. This is 
not surprising. If Developers are doing their job, they have kept in 
mine that the final product roust be in a form which the User can use. 
We 8ir.pl y note here that the Developers also need to consider that the 
developed product (prototype, model) can be manufactured , where such a 
function is needed. - To know what are the Production constraints, 
criteria, etc., the Developer needs communication linkages with the 
Producer. 

B. The Agency Role; Monitoring and Orchestrating System Linkages 

The mission of a funding agency in Development is in essence to 
insure that adequate Development is (or can) occur. Because Development 
is a system phenomenon, a major Agency responsibility is to monitor and 
orchestrate the system linkages. We emphasize the system nature of 
Development simply because Development carnot be isolated from 
ProductiDH, Dissemination, Implementation, and Utilization. 

In a mature system, this role is likely to be relatively smaller, 
as there may have developed natural linkages which provide for a high 
degree of self -monitoring and self -orchestration. 

Contrarily, in the uncertain conditions of an immature system, 
monitoring and orchestration may be a significant Agency role. 

II. niR EDUCATIONAL CONTEXT OF DEVELOPMENT 

1. The Lack of a Production Function in Education 

As we have already noted briefly, in the educational j^text, 
Development often does take on aspects of Production--simply because 
a separate Production stage and apparatus often does not exist. 

Mien no mass production is involved, the developed prototype may 
indeed be the sum total of Production. Then the issue is simply one 
of informing Users of the prototype. 

In other instances, the Production phase may occur at several 
unrelated points in time (instead of over a single, continuous point 
in tinoj. The prototype or model is put into the system, in the 
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literature, In ERIC--for retrieval by Users whenever they wish.. 

Of course, there are types of outputs (games, books) for which 
there nay be a distinct Production stage. Here the issue to bo con- 
.^iderc'd is: who is willing and capable to do Production? 

In cases where the Development work has been done by a commercial 
firm, the carrying out of Production activities is usually not a 
problem. Commercial firms tend to havfe well developed Production 
capabilities. And their right to control the nature of the output in 
its final form cannot really be challenged given their joint role as 
both Developer and Producer. 

However, when the Developer is government -funded, questions about 
commercialization in the Production stage arise inevitably. Production 
considerations may suggest major changes in the Developer's prototype 
if Production is to be carried out on a sufficient scale and/or if it 
is to be Icept within specified costs. In such instances, the Developer 
will hiVC t: reconcile his desire for Production on a scale that would 
permit widespread DLsseroination and Utilization, with his desire to 
maintain the integrity of his prototype and to control the character of 
the final product. This is clearly an issue worthy of some attention, 
but one that we cannot deal with here. 

Mention should also be made of the difficulty encountered occa- 
sionally when commercial Producers acquire the rights to a larger 
number of Development outputs than they can readily produce over the 
short run. The consequence here may be that the product's dissemination 
is delayed for some time (if not, perhaps, forever). This potential 
problem suggests the need for considering the possibility of agreements 
with commercial Producers that entail forfeit of the Producer's rights 
to a product if its Production and at least the beginnings of its Dis- 
semination arc not accomplished within a specified time period. 

For a variety of reasons, a product may not be commcrcialized-- 
either by intent or by default, or for lack of interest from commercial 
Prod.:.vrs. Where Production activities of some kind or other arc called 
for n Tho very nature of the product, then the question hucoinfs : if 
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not coamercial Producers, who w^^ do Production? There are a number 
of possible answers. Perhaps the Developer will do Production per- 
haps even the User or various kinds of intermediary organizations created 
rj link Producers and Users. The character of that Production may well 
vary depending on who in the system actually carries out the Producaon 
work. 

2. The Development Knowledge/Tech nology Base 

There seems to be general agreement that the educational Develop- 
ment knowledge/technology base is weak» that its quality is generally 
poor. E^i^ when one can point to some highly valuable Development 
knowledge or some highly useful Development technology, access to 
quality information of this kind is difficult: 

a. Developers do not know what other Developers have done or 
can do. 

b. There is little codification of Development knowledge/ 
technology— the kind of codification to which Developers 

I 

could turn and on which they could rely. There is an 
absence of "handbooks." Lacking too are distinct, 
discrete. Development-relevant categories to facilitate 
information searches. 

c. The transforms or linkage mechanisms between stages of 
the educational R/DQI process arc weak (e.g., there aie 
so many unknowns in education that Research findings do 
not readily get translated into Development products; 
almost equally ambigious may be the linkages between 
Development and Production or Dissemination) . Considering 
that nevolopment is a continuous process across various 
R/D5I stages, such transforms or linkages are quite 
important 

In light of the above, quality con:ro1 becomes a key issue for 
educational Ucvelopmenr . Further, since the quality control needs to 
be continuous throughout and even beyond the Development stage, there 
is a strong logic for brea'inR up the procurement of Development into 
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sta^s. As an example: 

^tage 1: The Developer is asked to present NIE with a concept- 

^ for development of a product. 

Stage 2: The Developer is asked to create and pilot test a 
prototype . 

Stage 3: The' Developer is asked to develop a scaled up version 
ready for large-scale Dissemination, 

At each stage, the Development process could be reviewed, re- 
fined and even stopped. The point is simply that NIE cannot afford to 
assume that adequate quality control will be provided throughout an 
entire Development cycle. 

This St age -by- stage quality control could be accomplished through 

either of two basic strategies. 

1. A single Developer could be funded for the entire Development 
process, but NIE would review the Developer's work at various 
stages in the process. 

2. Each stage of the process could be funded separately. This 
would allow NIE the option (at each stage) to continue with 
a single Developer or to open each stage to competition. 

The point to be emphasized here is simply that NIE should think 
about Development procurement along lines such as these. And if NIE 
is going to consider such strategies, a fairly active NIE involvement 
is required- -as contrasted, for example, with NIE simply providing 
long-term funding to a Developer who proposed what appears to be an 
exciting, potentially significant Development project. Such NIE in- 
volvement may seem onerous at times--but in educational Development, 
some active degree of NIE involvement is bound to be necessary. 

Finally, we must note that in education, the difficulty of 
developing "product stand \rds" is such that the very concept may be 
meaningless. The educational literature contains examples of some of 
the more rigorous Development organizations striving to develop 
"product specifications" i,iior to designing and testing prototypes. 
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Itowever, due to the "soft" nature «f the product in education, the 
weakness of the overall knowled^ base of the fiold, and the general 
resistance to developing and usirtg tight product specifications or* 
standards— either in the design phase or in the course of formative 
or summative evaluations-- there seetas little point at this stage in 
the maturity of the educational R/DSI Wsteni to invest in development 
of product "standards" as the solution the issue of quality control. 
More productive may be an investment in deigning Agency quality control 
monitoring processes that have wxthin thera\hc built-in potential for 
documentation and analysis of Development vprOicesses and the cumulative 
development of a stronger Development knowledgte/technology base. 

3. the Institutional/Personnel Base for Educational Development 
The institutional/personnel base for educational Development is 

weak. There are some strong Development organizations, but only 

relatively few, especially in comparison to other fields (e.g.. Health). 

There are personnel doing Development, but relatively few personnel 

trained in the Development process. 

The weak institutional /personnel base raises at least three major 

issues for NIE. 

A. Contractor Selection 

On the one hand, NIE must consider whether a potential contractor 
has the kind of professional capabilities and skills to handle Develop- 
ment throughout its various phases. 

Thus, selecting a competent contractor is a key role for NIE. How 
ever, the need for building an educational Development system is so 
strong that NIO must be equally concerned about ensuring an adequate 
Development institutional and personnel base from which competent 
contractors can be selected. In many cases, this will mean that NIE 
will have to take the initiative in seeking out and developing con- 
tractors (as contrasted to simply responding to field initiation or 
selecting from among those organ izat ions will in*; to invest resources 
in responding to competitive prociircmonts) . 
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B. Quality Control of Process Instead of Product 

Given the weakness of this institutional/personnel base, and 
the difficulties inherent in either devising or using meaningful 
product standards for quality control of the Development function in 
education, ^lE will do better po attempt to exert quality control 
over the Development' process rather than attempt what are likely to 
be fruitless efforts to control directly the quality of Deve.lopment 
products . NIE can exert some control over the Development process by 
determining how the contractor is going to approach a Development 
project, who is going to be involved, etc. 

C. The Locus of Quality Control 

Quality control may be provided in either of two ways. 

In a mature system, there will likely be a large degree of effec- 
tive field-initiated quality control. 

In an iuonature system (such as education), however, effective 
field-initiated control will tend to be missing. Tlius, quality control 
mechanisms will need to be set up by the Agency. (An exception can and 
should be made in those few instances where the Development institutions 
are able to and do exercise a strong degree of self-control over 
quality.) 

In the educational context, quality control could be provided 
directly by NIE; or NIE could procure the services of other organizations 
for quality control; or, as the field matures, NIE could gradually 
facilitate the development of field self-control over quality. 

Wc must further note that the locus of quality control will tend 
to shift through the various phases of the Development process. In the 
initial phases, NIE must play an important role — e.g., with Irespect 
to Need Identification, how Development will be done, who will do it, 
etc. As the Development pAaccss moves towards final ization of a 
developed product. User involvement will tend to be more important than 
NIE involvement. Thus, NIE should design and provide the mechanisms 
that will gradually shift the locus of quality control from NIE to Users. 
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D. Development in the Quasi -Public and Private Sectors 

Finally, we may note some differences between Devolopmt;»it in 
the quasi -public and private sectors. 

IVhen Development is done in the commercial (private) sector, there 
is in the long run a built-in quality control mechanism: if the product 
is not good, it is not likely to sell for long. This is especially 
true in fields where there are sophisticated Users. Further, in the 
commercial sector, the Development/Producfion/Dissemination phases axe 
integrated. The danger here is not so much that of selling a low 
quality product as it is selling the User products he does not need. 

In contrast, as we have been noting, when Development is funded 
by the government in the quasi-public sector, quality control cannot 
be assumed, nor is the Developer automatically linked into the total 
Development/Production/Disscmination process. 

4« Development Processes in Education 

Two aspects of Development as a process merit particular attention 
because of their potential implications for NIE behavior with respect 
to the field. 

A.. Need Iden ification: Program Definition/Project Selection 

1. So Many Needs 

It must be understood that especially in education, there are 
far more needs than can be met at any one time. Tliis fact necessitates 
having some form of project or program selection process. Further, 
given the limited resources of NIE, the major criterion must be cost/ 
effectiveness, with effectiveness being seen in system terms of multi- 
purposcs and of portfolio effects. 

2 , M ultl-Purpoies/System Impact 

Any particular procurement should be examined not only in terras 
of its stated purpose, but also in terms of additional ways the project 
might (or could be used to) impact various parts of the educational 
systci'i. For example, the project could have the additional impacts oC; 
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a. building the capability of the Deveioper; 

b. creating linkages; 

c. helping the User during Implementation and Utilization; 

d. impacting the educational Development environment by 
helping people see educational Development in a better 
light. . 

Thus, a premium should be placed on those projects which not only 
have important manifest purposes but ^ich also can provide multiple 
additional impacts upon the total educational ystem. 

3. Portfolio Impacts • 

Similarly the cost/ effectiveness of a given project should be 
evaluated in terms of its overall impact upon and within the total 
portfolios of projects and pr , -ams at NIE and at the various Development 
institutions. 

For example: 

a. How many Development products (and what kinds) can be 
disseminated without overloading the User? Over what 
period of time? 

b. Would it be bettor for NIE to concentrate its resources 
in a single area in order to obtain synergistic and 
multiplier effects, in contrast to scattering its 
resources among many "good" but unrelated projects? 
Such a consideration would also suggest the„need to 
examine how other (non-NIE) sources of funding were being 
used, and how funding might be coordinated across funding 

' agencies to obtain synergistic and nultiplier effects. 

c. To what extent would a given project tend to tie up a 
significant portion of an institution's resources? 
For how long? 

d. What would a particular project do to the balance between 
high- and low-risk projects? Short- term and long-term 
projects? What kind of balnnce docs NIF: want to have? 
What kind .should a given institution have? 



ERJC U!) 



84 



Thus, consixierdble emphasis should bo placed on those projects 
which fit best ittp the total portfolio of projects. \ 

Other illustrations could oa ily be given of multi-purpose and 
portfolio impacts. The point is simjily thatrinroject selection should 
be considered not just from the viewpoint of ajsingle project in 
isolation but also from the viewpoint of total system impact. 

4. NIE and Field Roles in Project Selection 

Viewed from this above perspective, project selection is not merely 
the evaluation of proposals. Even more significantly, project selection 
is a key element of system orchestration, in Development but also (of 
course) In other R/D§I system functions. 

Further, from this perspective » it becomes possible for NIE to 
think of potential divisions of roles between NIE and the field. 

Specifically, the field should be involved primarily at the 
point of proposal evaluation, while NIE must take responsibility for 
project, and system orchestration. 

The field should be in the best position to evaluate whether an 
*innovation is "goou" or "bad," at least in terms of such key criteria as: 
relevance to User need; User capability to implement and utilize; state 
of the art; User climate for acceptance; etc. Parenthetically we may 
note that to the extent NIE lacks education professionals on its staff, 
NIE will be dependent upon field evaluation. This dependence is not 
an issue hero. By allowing the field to provide evaluation from its 
^perspective and capabilities, NIE is freed from some unnecessary 
additional burdens and can focus its resources on: a) the evaluation 
of proposals in terms of orchestration and total system impact, <tnd 
b) doing the actual orchestration. At the same time, we do not want 
to downplay the importance of having education professionals on the 
Institute's staff. They would seem to be needed to help give an f^w^ 
cation perspective to NIE's orchestration role. 

The field cannot really bo of much help ii. making orchestration 
and total system impact evaluations simply because they are not likely 
to be in touch with (04^ even know about) all the elements of the NIC 
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Thus, considerable emphasis should be placed on those projects 
which fit best into the total portfolio of projects. 

Other illustrations could easily be given of multi-purpose and 
portfolio impacts. The point is simply that project selection should 
be considered pot just from the viewpoint of a single project in 
isolation but also from the viewpoint of total system impact. 

^ 4. NIE and Fjeld Roles in Project ^dl'ection 

Viewed from this above perspective, project selection is not merely 
the evaluation of proposals. Even more significantly, project selection 
is a key element of system orchestration, in Development but al (of 
course) in other R/D^I system functions. 

Further, from this perspective, it becomes possible for NIE to 
think of potential divisions of roles between NIE and the field. 

Specifically, the field should be involved primarily at the 
point of proposal evaluation, while NIE must take responsibility for 
project ani^ystem orchsstration. 

The field should be in the best position to evaluate whether an 
innovation is "good" or "bad," at least in terms of such key criteria as: 
relevance to User need; User capability to implement and utilize; state 
of the art; User climate for acceptance; etc. Parenthetically we may 
note that to the extent NIE lacks education professjionals on its staff, 
NIE will be dependent upon field evaluation. This /dependence is not 
an issue here. By allowing the field to provide evaluation from its 
perspective and capabilities, NIE is freed from some unnecessary 
additional burdens and can focus its resources on: a) the evaluation 
of proposals in terras of orchestration and total system impact, and 
b) doing the actual orchestration. At the same time, wc do not want 
to downplay the importance of having education professionals on the 
Institute's staff. They would seem to be needed to help give an edu- 
cation perspective to NlR's orchestration role. 

The field cannot really hv of much help in making orchor»trat ion 
and t.-t.»l system impact evaluations simply because they are' not likely 
to in touch with (or even know about) all the clemonts of the Nil: 



portfolio; to know which institutions need building; what funding is 
available from what sources for what projects; etc. In other words, 
the fl.eld has neither the system information nor the system perspective 
to^ke a total system, evaluation or to take responsibility for or- ' 

chestration. ' . 

Finally, wo may note that the field's lack of a total system 
perspective may well be a strong point in favor of NIE permitting the 
field to play a major role in evaluating individual proposals. Since 
the judgments of NIE personnel are likely to be affected by their con- 
cerns about system building, etc., evaluations of individual proposals 
by the field provide the Agency with a rather different set of evalu- 
ative judgments to input into the project selection process- -judgments 
based on the merits of individual proposals alone, unhampered by the 
other kinds of concernis that would arise inevitably from application of 
a total systcir perspective. 

Potential Equivalency of Alternatives 

In Development (as elsewhere to <*omc degree), it is not obvious 
and clear that one product will be of more benefit or is more necessary 
in the long run than another product. Innovation rarely comes from a 
once-and-for-all "great breakthrough." Rather, significant innovation 
generally results from a series of small improvements, and such im- 
provements may result from a myriad of alternative approaches, methods, 
project:t, etc. 

Thus, NIE will always be in the position of making cost/effective- 
ness orchestration judgments , not only between proposals, but between 
the need to orchestr^itc the field, on the one hand, and the value of 
a given innovation, on the other. And NIE will not^have the luxury of 
certainty about its judgments. This does not m'^an that NIE should avoid 
the responsibility of making cost/effectiveness judgments by relying on 
others (e.g., a panel of "experts" from the field). As wo have already 
noted, the field canno t orchestrate. Rather, the Agency staff, after 
obtaining the best information it ean, must simply a'- cpt the rcsponsi- 
biUty to make (and live witli) -rchcst rat ion judgmonts. 
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6. The Extent of NIB Influence in tho Field 

In terms of funding, NIE is only one of many sources which fund 
education or education-related Development. NIE is not even the 
largest funding source. Further, NIE does not have the power or status 
of such other funding agencies as the National Institutes of Health. 

Nonetheless, since NIE is the lead agency for Research and Develop- 
ment in education, its orchestration, role is vital. Futher, even 
though NIE, may not have a large direct influence on the field, NIE 
can have a significant indirect impact on education through its orches- 
tration role, by tailoring its efforts carefully in terras of such 
criteria as: knowing what is and is not being done; knowing where the 
strengt>^ and weaknesses of the system lie; knowing where it can and 
cannot help. Indeed the combination of NIE*s relative smallness and 
its having the lead responsibility make its orchestration role even more 
important . 

7. Appropriate Evaliiators 

Selection of personnel to evaluate proposals, projects and products 
is a key element for an overall NIE Development project selection strategy. 
Determining at what point (during the overall Development process) each 
e valuator has an appropriate role is also important: 

1. Users have a valid evaluation role at three main points: 

a. early in the Development process, to identify their 
needs; 

b. at the point of proposal evaluation, to help NIO 
evaluate the specific merits of individual proposals; 

c. later in the Development process, as products bejjin 

to enter tho deve loped stage; and then as products arc 
implemented and utilized, so as to provide corrective 
feedback to Developers and/or Producers. 

2. Dcvolopers should have a role early in the process to help 
identify what projocts should he workfd on, especially to 
provide some input into derisions about what is possible 
given the Development state of thi* art. 
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Researchers will play a very minor rolo in the evaluation 
process. Where the state of the art leads to a very large . 
degree of uncertainty, Researchers might be able to provide 
some insight. 

NIC has two basic evaluation roles: 

a. NIE must evaluate proposals from the total system 
impact perspective (multiple purposes and portfolio 
impact) . 

b. NIE must also select the cvaluators from the field, 
both in terms of individual capabilities and perspectives 
and of providing balance in the field evaluation. For 

' example, if there is to be an evaluation panel, it is 
NIE's responsibility to determine how many and what ^ 
Users, Developers, Researchers; at what different points 
in time; organized in what ways; etc. In other words, 
the use of field evaluators should be a carefully thought - 
through and planned process, one which can vary according 
to situation. 

Sources of Need Identification: Practice v. Technological 
Opportunit y 

Thus far we have been emphasizing practice-based needs--i.e., 
the needs identified by Users, which are based on their everyday ex- 
perience. It is also important to consider needs which are technology- 
based- -i.e., when it becomes technologically possible to develop a 
product, consideration must be given to developing (and in some cases 
even promoting strongly) that product. 

The issue now becomes: IVhat is a proper portfolio balance between 
needs as currently understood by the fiold and responsiveness to what 
has recently become techno log\val ly possible. On the one hand the 
"bread and butter" type of Development is generally based upon User 
identified needs. On the other hand, it likely that the accumulation 
of Ucvelopmont and Research knowlcdgo will at times make possible the 
development of a very significant product. Thus, NIU will want to 
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orchestrate a portfolio that permits strong technological oppor- 
tunity t>7»os of Development work to»be supported. 

The implication for the NIE-orchcstrated selection process is 
that Users will predominate in the evaluation of User-based needs, 
and Developers (with some help from Researchers) will predominate 
in the evaluation of technological-opportunity-based Need Identi- 
fication. 

9. Interrelationships and Orchestration; Some Implications 
To summarize some points we have been making: 

1. Consideration must be given to potential synergistic 
effects across projects that may result from develop- 
ing a portfolio of interrelated projects— as contrasted 
to a lack of synergistic effects from an unconnected 
set of unrelated projects. 

2. Consideration must be given to work that is supported 
elsewhere (e.g., by the Office of Education, the 
National Science Foundation, other non-government 
foundations). Here the relevant issues would be (for 
example) : 

a. How can the various funding sources pool resources 
and orchestrate efforts to develop synergistic 
effects across projects? 

b. How adequate are the non-NlE sources of support 
in a particular area? Are they such as to allow 
NIE to give its attention and support to other 
areas? 

^' Two Basi c Operat ing Modes in l-ducay onal D evelop ment 

Basically, there arc two rather different operating modes 
through which the Development function is carrlevl out in 



99 



education— De\telopment work as it is generally carried out 
in sp€tciali:ed Development organization, and Development 
work as it tends, to take place in practice-based (or 
practice-related) settings. 

1, Specialized Development Organizations : 

The Development function as it is carried out in pur- 
suance of government contracts --primarily in federally- 
funded regional laboratories and R§D centers and in 
some of the non-ptofit and for-profit Research and 
R^D corporations— tends to adhere, at least in form, 
to the engineering model of Development used in indus- 
try. 

Development in these settings is an institutionalized 
specialty carried out by specialized personnel in 
specialized Development organizations or organizational 
units. In the best of these organizations. Development 
activities tend to be systematic and sequential, moving 
in a smooth progression from the prototype design that 
is the end product of the Research phase of R/D5I, to 
product or program Development in accordance with detailed 
specifications, to evaluation of .small field tests, to 
revisions, to larger field tests, to more revisions, to an 
* additional field test, etc. until the product performs in 
accord with prespecified performance objectives. Products 
go through successive generations of revisions, each a 
closer and closer approximation to the performance specifi- 
cations. Revisions are based on empirical field test data 
that are gathered systematically and analyzed rigorously, 
and the evaluation data are expected to provide the poten- 
tial User with information about the outcomes or effects to 
be expected from use of the product under specified Im- 
plementation conditions. 
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Clearly, there is a considerable amount of Develop- 
ment work carried on in these organizations that does 
not adhere faithfully to this relatively rigorous model 
of the Development function. But equally clearly, this 
does appear to be the model they are trying to use in 
plannirg and conducting their work. 

Development projects implemented in accord with this 
model tend to be l.'>rge->scale axiA expensive, involve large 
personnel pools and heterogeneous skill mixes, and require 
extensive cooperation between the organization developing 
the products and the school systems agreeing to serve as 
field test sites. The products themselves are often com- 
plex, consisting of many and varied modules or components* 
and often several forms of media as well as printed mater- 
ials. The management of these complex projects is often 
highly formalized, using flow charts and sophisticated 
management tools. 

There are some variations in pat urn depending on the 
nature of the products being developed— e.g. , products vs. 
change proces5es. But the issues of concern to managers 
tend to be consistent— e.g. , How much Research is needed 
prior to the Development work; how much Research can 
proceed parallel to the Development work? At what point is 
the product sufficiently developed to permit initial field 
testing? At what point has the product been tested suffi- 
ciently to permit Dissemination? What Dissemination, Market 
ing, and Implementation factors need to be considered 
throughout the desigii and developmenl phase? At what point 
does the responsibility of the Developer end— Development? 
Dissemination? Installation? Utilisation and Maintenance? 

These issues are to some extent common to the Development 
function in all sectors, but they take on particular signi- 
ficance in education. The weaknesn of the knowledge and 



technology base of the field nakes it more difficult to 
translate performance specifications into effective 
products. Outcones are much less predictable given: 
a) the nature of the interaction between User and product 
and b) the limited technical capability of U&ers to imple- 
ment complex innovations without substantial Bnplementation 
supports and/or assistance. Consequently, Development work 
in education requires a' far greater investment of^^me and 
money in the Research and Evaluation components of the 
RQD process, making Development costs high relative to 
practical payoff, a problem of particular importance 
considering the negative political climate in which edu- 
cational R^D appropriations tend to be made. 
Practice- Based/PracticerRelated Development : 

A rattisr different mode of Development is inherent in 
traditional ..pproaches to the design and development of 
instructional strategies and materials— as these activities 
have been carried out by classroom teachers, by curriculum 
specialists in SEAs^ L£As, and universities, by publishers,' 
and by the university scholars who' have on occasion par- 
ticipated in efforts to improve K-12 level cunlcula and 
instructional materials in their areas of specialization. 
We note in particular the significant amount of program 
design and development that has been carried out within 
LEAs in response to the availability of cat -.^orical 
funding from federal (and some State and other) jjAurces — 
e.g., ESEA Title I and Title III funding from the Office of 
Education. 

The Development approach used in these settings tends to 
be intuitive rather than data-based or grounded in in- 
structional theory. The focus of attention is generally on 
the content to be conveyed rather than on conceptions of 
how teachers go about providing instruction. Field-testing 

1 ^ If . 
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is non-existent or minimal, hliatever evaluation is 
carried out tends to focus on the face validity (e.g., 
expert judgments, teachers* subjective perceptions and 
reactions) rather than measured effectiveness in achieving 
prespecified impacts. There may not be a systematic 
testing -program or rigorous analysis of empirical data. 
In comparison to the Development mode that characterizes 
specialized Development organizations, there is less likely 
to be a systematic cyclical test and revision sequence. 
Even where some evaluation and revision does take place 
there is less likely to be extensive recycling, and it is 
highly unlikely that evaluations would be made in terms of 
a product *s effectiveness in achieving prespecified measur- 
able objectives. 

There are other significant contrasts between the two 
Development modes. In comparison to the pattern in special- 
ized Development organizations ; Development costs in 
practice-based (or practice-related) settings tend to be 
relatively low. The personnel involved are relatively 
few (e.g., one teacher, a few scholars or curriculum, 
specialists, etc.) and whatever skill mixes are present in 
a Development team tend to be relatively homogeneous. 
Management is generally, informal and highly flexible. 

Within the practice-based/practice-related mode, two 
important variants can be identified, depending on whether 
or not the product or program of interest has been developed 
with the intent of wide-scale dissemination in mind. Where 
textbooks or materials packages are being developed for 
large-scale, nationwide dissemination, an effort is usually 
made to include Implementation supports in the form of 
teachers' guides, tests, etc. Where materials are developed 
locally within the User system for use by a single teacher 
or group of teachers in a single school or District, far 
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less of the Ir.pler.entation process is committed to print 
or media presentation, the state of "development" of the 
materials or strategies for use outside this small gioup 
remains inadequate, and either the locally developed in- 
novations are not dis;5eminated at all or they are dissemin- 
ated but have minimal success elsewhere because Development 
work was not carried far enough to permit the materials to 
be implemented easily and effectively by others. 

The exemplary practices or programs that have become, the 
focus of so much attention in recent federal and State 
Dissemination programs fall within this practire-based mode. 
Where exemplary practices or programs are neither designed 
in a form that permits them to be generalized to other 
schools or Districts, nor packaged in a form that permits 
disseniinaxion, these practices/programs may provide poten- 
tial inputs for the Development function as this is carried" 
out in the specialized Development organizations. We shall 
return to this point shortly. 
Strengths and Weaknesses of the T\v o Basic Modes 
Given that Development work in education tends to be carried • 
out in one or the other of these two basic modes, our first 
concern must be to identify the strengths and weaknesses of 
each mode. We will then be in a position to identify several 
issues with significant implications for NIE behavior, 
a. Specialized Development Organizations : 

The specialized Development organizations in the edu- 
cational R/D5I system have (at least potentially) several 
basic strengths. They are generally largo in scale and 
tend to be staffed by sizeable numbers of skilled personnel 
(quite possibly) at the "minimum critical mass" level. 
These personnel arc more likely than the practice-based 
and rracticc-related Developers to be knowledgeable about 
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the Devel9praent state of the art. These organizations 
tend to have the capabilities needed to undertake large- 
scale projects and are generally able to carry out all 
steps of the Development process. 

These organizations also have several potential disad- 
vantages'. Because they are external to the User system, 
they may tend to be isolated and insu. at^d from User needs- 
working, as it were, in an "ivory tower" setting. Dis- 
semination, and Implementation may be problematical because 
of inadequate linkages to the User population and/or to 
intermediary organizations,: because their products may 
not be sufficiently relevant to User needs, and because 
the User may react to these externally developed products r 
in terms of the all-too-familiar "not invented here" 
syndrome. Given the high costs of rigorous Development 
work in these specialized organizations, the relatively 
limited utilization of their outputs to date in User set- 
tings, and increasing recognition of the extensive amount 
of local innovation that exists in some parts of the User 
system- questions continue to be raised about the relative 
efficacy of this Development mode. If this mode is to 
remain viable it would seem essential for NIE to relate to 
these organizations in a manner that strengthens their func 
tioning. 

b. Practice-Based/Practice-Related Development ; 

The obvious advantage of the practice-based/practice- 
related mode is its general closeness to real User needs 
and problems, especially ii; those cases where the Developer 
is the User or is at least part of the User system. 

However, several disadvantages of this mode are also 
, obvious. For example: 

1. This tends to be an inefficient and at times in- 
effective mode. Since the personnel carrying out 
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this kind of Development work tend to be part-time 
Developers t best (and often people who may carry 
out Developi. nt work only once or twice or very few 
times) --personnel who by training, background, and 
commitment tend to be identified with other kinds of 
activities— there is limited likelihood of these 
Developers being very aware of the Development state 
of the art. Thus, there will tend to be much "re- 
invention of the wheel"— duplication of effort, etc. 
Further, most of the Developers who carry out this 
type of Development work lack both the skills and 
time needed for sophisticated and complex develop- 
mental activity. (One important exception here are 
the Production personnel who generally work for 
textbook publishers. But even in the publishing 
example, the university scholars or education 
practitioners who may do the bulk of the conceptual 
and substantive, as opposed to editorial and Pro- 
duction work, are likely to fit the above description.) 

2. Even when the Development output appears to be high 
in quality, there is generally little accompanying 
empirical evaluationJiata to serve as the basis for 
making quality control judgments or Adoption/ 
Adaptation/Implementation decisions . 

3. IVhen Development work is carried out within the User 
system— whether in classrooms, curriculum committees, 
teacher centers, LEA or SEA off ices-- the output (a 
curriculum, a set of materials, etc.) is being 
developed for a particular local component of the 
User system and is generally not automatically input 
into some broader Dissemination system able to make 
it accessible to a larger population of potential 
Users. Thus, there arc rather substantial problems 



ERIC 



96 



encountered in identifying and disseminating high 
cAiality developmental work carried out in these 
' ettings . 

4. There is one other significant problem related to 
the fact that this kind of output is generally 
being developed for only a local component of the 
User system— by a User-Developer for personal use, 
or by a Developer who can interact personally with 
a relatively small number of Users (e.g.^aa LEA 
curriculum coordinator, or a teacher working with 
other teachers in her school or District). More 
often than not, the kind of Implementation informa- 
tion that a User-Developer is likely to carry around 
in his/her head is not elaborated in, print or through 
media presentations. For this is the kind of in- 
formation that the User-Developer generally feels 
need not be committed to paper for personal use, and 
can be readily communicated orally in interpersonal 
exchanges with other Users in the school or District 
with whom he or she may share the Development output. 
If this kind of output is to be diss-^rainated more 
broadJy it must.be packaged properly--a problem usually 
beyond the time and skills (and probably the interest) 
of the individual Developer-User. 

c. Issues 

In relating to the specialis:ed Uevclopment organizations 
in the educational R/D5I system, the key issues for NIE 
to consider would seem to be: 

1. the extent to which NIE should specify the substance 
of, anJ/or provide direction to, their Development 
activities; ^ 

2. the extent to which emphasis should be given to 
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system building and/or- rebuilding; and 
3. how to overcome the Dissemination woaTcnosses in- 
herent in this operating mode based in settings 
external to the User system. 
In relation to practice-based/practice-related Develop- 
ment work, the key issues foi* NIE would seem to be: 
, " I. to what extent these outputs should b© validated, 
and what mechanisms might be most appropriate ^or 
whatever degree of validation is seen as needed; 
• 2. what mechanisms might b- most appropriate for 
identifying high quality materials, programs, 
practices, ^tc. developed within the User system; 

and ' .. , 
3. Khat strategies might be most appropriate for 

suppler.enting the efforts of the Developer-User at 
various steps in the Development process that are 
not likely to be carried out adequately (or at all) 
without the intervention of some State or federal 
agency or some intermediary ofganization (e.g., 
the packaging. Production, and Dissemination of 
exemplary programs or practices created by User- 
Developers) . 

It is woth noting that Dissemination is ann'ssuc for 
both of these two basic modes of educational Development. 

Given that these ar£'^^'° ^^^^ significant modes o^' 
educational Development, the overarching appears to bt . 
on the one hand, how to capitalize on their strengths 
while on the other hand, how to build into the s>'st<'m 
countervailing forces to overcome their v;oaknossc<;. 

4. The ME Rol e 

It ::in be seen from the above del i neat o<^ issue that 
:;IH rust accept r<-"..poiis lb i li t y for onliestrat i i)<; the 
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linkages- .ply because the need is ther^ and thewe is 
no other agency with the authority (or perhaps too the 
inclination) to do this orchestration. 

In relatioi), to the large-scale Development organiza- 
tions » there is a i ed to develop the linkage between 
these organizations and Users. NIE should be directive' 
in developing and controlling the^ linkage process . 
That is, NIE should insure that the process- leads the 
Developer both to develop products that are needed and 
to disseminate these products. This is related to our 

9 

earlier point about the needed User role in Development 
program definition and project selection. 

In relation to the practice-based mode of Deyelopment, 
the needed NIE^ role would seem to be to develop and/or 
facilitate mechanisms wh5ch would permit the identifi- 
^cation of exemplary practices and other outputs which 
could and should be generalized^ packaged and dissemiliia- 
ted (and probably validated as well). The question now 
arises: who should begin the process by identifying 
exemplary practices, etc. for input into more formal- 
ized and rigorous developmental activity, e.g., pack- 
atging and validating? On the one hand. Users could 
identify such practices and bring them to the attention 
of some other (probably intermediary) organizations. 
On the other hand, intermediary organizations could 
bear the brunt of these ^'search and discover'' opera- 
tic^ . 

Once such exemplary practices have been identified, 
they must then be recast in forms that are generalize 
able to other settings, and then, t!icy must bo packaged, 
produced, and disseminated, r 

Wliere organizations with the needod capabilities 
already exist in the field, NIlPs role would seem to be 
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to facilitate their work (and where possible their 
growth) by providing resources ond/or making it more 
feasible In other ways for them to function better and 
to grow. 

However, if the needed organizations do not exist 
(OT^are too few), NIE's options would seem to be to 
either "buy" or "rent" the needed capabilities. The 
"buying" strategy would involve a direct attempt to 
create insitutions that would become a permanent part 
of the educational Development system. A "renting" 
strategy would entail the temporary , purchase (there- 
fore, the term "renting") of services from organizations 
that already possess the needed skills but are external ,, 
to the educational R/D5I system. In this case, a 
"ga?" in the system is "filled," but only temporarily, 
on a project-by-project basis. The advantages of the 
••renting" option are that the packaging, production, 
etc. can be done quickly, NIE can b^ rather directive 
about what is to be done, and the funding need be for a 
specified (short-term) period only. The disadvantage 
is that this option does not increase the overall long- 
term capabilities of the educational Development system. 
We shall return to this point later in our analysis. ^ 

Thus far we have discussed possible NIE roles and 
options vis a vis Development orgarii zai.ions and practice- 
based Developers as separate, unrelated courses of 
action. However, if NIE were to pursue a "buy" strategy 
for packaging exemplary programs, practice-based 
Developers and Development organizations could be linked 
in such a way as to improve the ovoroU, long-term 
capabilities of the educational Development system. 
Here, the role of NIH could be to create special Hnk- 
ires between Users and 0 jvelopnicnt orgnnications such 
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that the flow of exemplary practices into the 
Development organizations for packaging and dissemina- 
tion would become a normal and permanent part of the 
Development system. Initially, we would anticipate 
that the bulk of the exemplary practices that would 
flow into the Development system for packaging and 
dissemination would be identified by intermediary or- 
ganizations specifically assigned the task of identi- 
fying exemplary programs, practices, approaches, etc. 
operational in the User system. Over time, however, 
we would anticipate an increase in User-initiated 
communication of exemplary practices to the Develop* " 
ment organizations. This linka^-^TKf 'communi cation 
flow might be strengthened over time as the outputs 
of these Development organizations become increasingly 

9 

User-relevant and high-quality. Under such conditions, 
we might hope for a greater User awareness and appreci- 
ation of the role of these Development organizations. 
Thus the real issue is: 

1. whether Nin wants U insure that many exemplary 
practices are identified, packaged, and dis- 
seminated quickly (and therefore might choose 
the **rent*^ option) ; or 

2, whether instead NIE prefers to build slowly a?)d 
facilitate development of capabilities and 
linkages that will become permanent and expand 
the capacity of the overall system (and there- 
fore mijiht chooso instead to function in a 'U^uy'' 
mode with built-in system building potential). 




III. I.VPIICATIONS FQ» NIE BEHAVIOR 
^ 1> Procurement Behavior 

The key theme throughout this analysis of educational Development 
has been that NIE is not simply a "channel" for funds. Rather, NIE has 
a major orchestrating role because the context calls for this type of 
role, the need is so great, and there is no other institution available 
to carry out this role. Further, orchestration implies not simply a 
balancing of types of products or programs, but also'- -and more important- 
ly- -a .balancing of purposes. As noted in our introductory chapter, 
when NIE makes a procurement, it may procure: 

1. a product or a program; 

2. system building; and/or 

3. some change in the environment of the educational 
system. 

;.7.cc>.?r it intends to do so or not, whenever NIE makes a procurement, 
that procurement may affect one, two, or all three of the above purposes. 
Thus, for example, a procurement made with the manifest purpose of 
permitting the rapid design of a given product may achieve that parti- 
cular purpose and have little if any discernible systemic impact. Or, 
the procurement may achieve the manifest product-design purpose while 
at the same time also strengthening (or weakening) thclQ»ge*fc«Qn 
cap.ibilitic-s Ox a particular part of the cducatyrltal^j^Qhpmcnt system. 

A. The " Wmt" of Development Procurement wxb<^ t.rati^n 

Throughout this analysis, we have noted a numljcr of specific 

items which NU; can .^u should orchestrate through its Development 

funding policy. ^ • < -nnr ^o; 

i 

1. obviously, a ';p'.H:ific, relevant, und usable Development 
^ product; 

2. system linkages (e.g., between Users and Developers, be- 
tween Devclopevs and the Development stale of the art); 

3. balance amonj> ami between multi-purpor.es and portfolio 
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effects; 

4. coordination and synergistic effects between NIH and 
other funding agencies and/or combinations or projects; 

5. system biJTlding and rebuilding. 

B« The "How" of Development Procurement Orchestration 

We have noted a number of options or alternative strategies 
concerning the methods of orchestration. 

1. NIE can operate in a consensus building mode, i.e., pro- 
viding the mechanisms whereby field personnel can come 
together to reach agreement as to key issues, directions, 
etc. Basically this is a process of orchestration through 
the use of field personnel for inputs to key planning 
decisions. However, we note that NIE must orch' strato 
the selection both of which field personnel and which 
mechanisms are to be utilized, and in relation to what 
issues or concerns. 

2. NIE can choose to orchestrate through the proper selection 
and placemen t of advisors in relation to specific aspects 
of the Development function. That is, NIE can orchestrate 
procurements through decisions about who will be involved 
in Need Identification, in various stages of quality 
control, in making decisions about field testing and pack- 
aging, etc. 

3. Orel cstration may be carried on through staging the nrocure- 
ment proccss--eithcr by using different contractors for 
different stages of the Development process (e.g., differ- 
ent contractors for Need rdontificatJon, designing, field 
testing, etc.); and/or tuiuling only one stage at a time, 
reviewing afm- each stage, and th< n continuing, modifying 
or torminatinji the process. 

I. NIH has choices to make among the strategies of facili- 
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tating, buying or renting. Wc will discuss this more 
fully below. 

Regardless of which combination of the above options 
NIK chooses, there are two aspects of orchestration method- 
ology which our analysis has emphasized. These are: 

5. Orchestration requires a monitoring process. Thus, moni- 
toring becomes a critical NIE activity that has to be 
continuous and ongoing. 

6. Given th« nature of the educational R/D5I system as a 
whole and of the educational Development system in par- 
ticular, orchestration of proce ss is likely to be more 
fruitful (and even feasible) than directive control of 
products . 

C. Ficilitatin,i. Buying or Renting 

ME can choose amonp facilitating, "buying." or "renting" 
strategies, both within projects and across its total portfolio 
of Development projects. Wc discussed these options earlier in 
connection with the ^xclaiTin^. production, and dissemination of 
exemplar practices deveiopea within the User system. The same 
points could apply equally well to other kinds of Development 
activities. 

A.-, nnted oarlicr, though, it should be underscored that the 
choice among these options must be based in part on the kinds of 
tapabilUies that do or do not exist at a given point in time 
withii.il the educational R/DU system. ^ facilitating strategy is 
possible for NIE only where the needed capabilities exist to 
some degree or other within the educatic-al R/DM system. If the 
needed institutions do not exist (or are too few), the facilitativc 
strategy is irrelevant. Thus, NIH would then have to create the 
neoded parts of the system, cither on a permanent basis (buying) 
. r on a temporary basis (rmtin^). 

Hach of these strategies call for a somewhat different managc- 
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laent moUc by NIE. IVhere a facilitating strategy is possible, the 
collaborative mode of management is most appropriate. The buying 
and renting strategies, in contrast, call for and permit a more 
directive mode of management by NIE, with renting requiring/ 
permitting the most directive approach. 
Three points should he made here: 

1. Both buying and facilitating are system-building modes. 
Renting is not, though some system-building capabili- 
ties can be built into a renting scrategy. 

2. In general NIE should seek to build the educational 
Development system (though there are exceptions, as 

we shall note), and where possible it is best to build 
^ through the facilitation of existing institutions. 

3. NIE has been receiving some criticism from the field 
for behaving in a manner that some in the field view 
as overly directive. IVliile we, in this analysis have 
been emphasizing a process rather than a directive 
management mode for NIE, we must also noce that at times 
NIE may have no choice but to be directive. A facili- 
tative mode, by definition, is relevaat only when con- 
ditions in the field permit. When there is a lack of 
adequate capability in the field, it is this weakness 
which requires the taking of a directive approach. 

Some additional comments ild seem to be in order 
about the renting strategy. A. already noted, renting 
is not a system-building strategy. That docs not mean 
renting is by definition a "bad" or "lesser of evils" 
strategy. For one thing, the nature of Devctopment is 
such that there may bo a large number of specialized 
tasks. Where a task is highly specialized, and/or seldom 
used, renting may indeed be the "best" strategy. There 
is nothing; inherent in 'the Development process that xC- 
quires a I tasks to bo pcrforniod by organ i?.at ions within 
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the Developnent system of o particular sector. 

A second point is that renting can be very helpful 
as an interim strategy— either because it is simply 
necessary, or because at a given time NIE is not yet 
certain whether there .is or is not a long-term need for 
bunding a specific internal system capability. 

Nonetheless, where renting is used as an interim 
strategy, NIE must, recognize that renting can retard 
system development. If NIE wants to develop a particular 
internal, syste'^ capability in the long run, there is the 
danger of never starting the process of building simply 
because the products being rented look so much better 
than what initially could be obtained within the system. 

Thus, where renting is an interim strategy, there 
should be some mixing of renting and buying strategies. 
For example, the rental contract might mandate a certain 
amount of training for personnel within the educational 
Development system. 

Finally, even when renting is a valid strategy, there 
are factors which will tend to constrain NIE's willing- 
ness to use the strategy— e.g. , a high price, a large 
amount of sub-contractin|^, etc. 

D. Excellence in Development 

We should note here that while "Excellence" may be considered 
a desirable procurement objective, "Excellence" has a meaning in 
Development rather different from its meaning in Research- -a 
difference that significantly affects funding policies and decisions. 

In Research, "Excellence" is measured against the best the 
field can do working at the outer limits of the state of the art. 
Thus, in Research, "Excellence" implies creativity, the utmost in 
understanding a::i krowlcdgc, even "breaking through" or expanding 
the limits of t'-e state of the art. In Research, "Uxccllenco" Is 
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not measured against practical considerations such as .cost, time, 
operability, availability, usability, etc. 

In contast. Development "Excellence" is^ measured by these 
practical operational criteria and is not measured by the "outer 
limits" of the state of the art. 

For example i in Development there may be a choice between two 
potential contractors. The first can develop a product which is 
''good enough to do the job," and can do so at a reasonable cost 
and within a reasonable period of time. Further, to fund this 
organization would result in strengthening tlie overall Development 
system. By contrast, the second organization can develop a product 
which is potentially superior and certainly more creative, but 
can do so only at a much higher risk and cost and it may take much 
longer. Further, funding the second organization would not result 
in any overall or long-terra strengthening of the Development 
syster.. In Research, the wise policy would be to fund the latter 
organization. In Development, however, it may be a wiser policy 
to fund the first of these organizations. 

In Development, usability is the key criterion of Excellence* 
Usability, costs, time-frame and system-building would seem to 
be the criteria most appropriate for project selection. 

E. The Effects of Funding Policy on Development Institutions 

NIC should keep in mind that its funding policy toward Develop- 
ment institutions tends to mold the character of these institutions. 
It is the nature of institutions (particularly the more raission- 
oriontcd) to tend to become what their funding sources (on whom 
their existence depends) demand of thera in terms of kinds of 
products, kinds of skills, kinds of risks, etc. It is, in simple 
terms, a case of "He who pays the piper calls the tunc." This 
dynamic is especially true of Development organizations, for \ 
tiioy, in sharp contrast to Research organizations, tend to be j 
hblo to acquire iMpidly the skills roquirod by the typos of ' 
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projccti funded. For example, in comparison to Research organiza- 
tions. Development organizations tend to have a clearer division 
of labor, tend to acquire personnel specializing in less creative 
areas of work, which require less knowledge, etc. 

A related concern is Whether (and to what extent) NIE should 
fund Development organizations on a programmatic as contrasted to 
an institutional support basis. NIE has been grappling with this 
issue since the beginning of its existence. The tide now appears 
to, be moving back noticeably toward acknowledging the need for 
some degree of institutional support for organizations with 
which it has (or chooses in the future to have) a "spe.cial rela- 
tionship's—or at least some kind of balance between institutional 
and programmatic support. 

V.'e note this concern because (in contrast to Research or- 
ganiziatior.s) Developnent programs which are government -funded 
tend to allow very little room for interpretation or maneuvering. 
Mien a Research program is funded, there is a sufficient level 
of uncertainty involved to permit the Research organization to 
choose from among a variety of alternatives and still remain 
within the program' s guidelines. By contrast, a specified 
Development product tends to be defined more narrowly and the 
Developer's options are considerably more limited. 

Thus, for the overall health of a Development organization, 
some institutional funding would seem to be essential, whether 
this be 10% or 20% or some other figure. Otherwise, the creative 
aspect of that organization will be stifled and its character 
molded to a very large extent by the funding agency. 

F. Competitive Bidding vs. Sole Sourc e 

Kc must r.ow call attention to an issue which is, to an im- 
portant c!e^;r.?e, political. Government policy favors competitive 
bidding f.-r novcrnmont funding, and for very valid reasons. 
Howc\-.-r, the very concept of orchostrat i tm (ospccinHy in the 
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context of an imraature system such as educational R/D5I in general 
or educational Development in particular) implies that there will 
be a significant (probably predominant) need for sole-sourco 
t>'pe of funding. 

Orchestration requires the ability to control the various 
parts and aspects and purposes of system funding. By definition, 
competitive bidding takes away from the funding agency a signi- 
ficant part of its ability to orchestrate. Even when there is 
competitive bidding, the ability of a potential contractor to 
"fit into" the funding agency's orchestration efforts may have 
to be one of the criteria for the contractor's Seing permitted 
to submit a bid and/or for the selection of one bid over another. 
For ^example, if one of NIE's orchestration objectives is to build 
a few strong Development organizations (as compared to funding 
rar.;- '.veak institutions with little or no system-building effects, 
given limited NX E resources), a fully open competitive bidding 
policy for each individual product or project may well negate 
such NIE orchestration efforts and capabilities. 

G. Single vs. Multiple Institutions 

Thus far, our discussion may seem to imply the funding of a 
single institution to carry out a particular project. This need 
not be the case. There can be good reason to fund several organiza 
tions to work on a single project. For example: 

1. No one institution may have aU_ the capabilities needed on 
a given project. For example, one may be strong in Need 
Identification because of strong ties with Users--but have 
lesser product design skills. Rather than eliminating that 
organization from the project, an option could be to fund 
a second organization for the product design stage. An- 
other example could he a product requiring the perspective 
of different disciplines (e.g., psychology and sociology as 
ivcll as education). Under such ci rciunstanocs. Nil; might 
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choose to fund a collaborative relationship between two 
or r.ore organizations with the necessary perspectives 
and s k i 1 1 • . 

2. Collaboration between two or more organizations could 

have synergistic effects. 
The use of several organizatipns could take a number of forms. 
For e;!Lainple: 

1. Staging . Different organizations would be funded for 
different stages of the Development process, iere the 
role of NIE would be to orchestrate and facilitate the 
transitions between stages and organizations. , 

2. Collaboration . Several organizations would funded to 
work together on the same stage or stages of the Develop- 
ment process. Here, NIE's role would be significant in 
selecting the organizations and perh^i)s facilitating 
their collaboration. 

3. Su pplementing . Here, one organization would have primary 
responsibility but would need soi..e supplemental help in 
one or more aspects of the process. Here, the NIE role 
could be either to select an organization which has con- 
nections with "satellites," or to select the supplemental 
organizations. 

4. System building . Here, a weak but potentially valuable 
organization could be "paired" with a stronger organiza- 
tion. The "weaker" organization could thus gain experi- 
ence. Here, the primary NIE role is selection. 

5. Fundi n|i Coordination . U-hero other funding agencies cither 
urc or could be involved, the use of more than one Develop- 
ment organization might simply be a political necessity in 
order for funding to he coordinated. Further, the use of 
multiple Dcvelopir.o.it organizations could be seen by the 
var^o::? funding agencies ar. having a potential synersintic 
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benefit. 

Regardless of ne form, use of multiple organisations would 
ne:e5sitate. some form of orchestration by NIE. 

2. Non- Procurement Nit: Activities 

Tlius far we have tended to focus on the procurement ac- 
tivities of NIE, i.e., those actions which lead to the award 
of (and invovle Agency personnel in 'monitoring) grants or 
contracts for various R/D5I outputs. Indeed, if one were to 
analyze NIE's activities solely in terms of its expenditures, 
it might well seem that the Institute's role is defined 
largely in terms of procurement activities. 

However, there are various v otheA Itinds of activities 
which NIE could initiate, or facilitate, and we have mentioned 
some of these at various points in our analysis. Examples 
of tnose non-procurement activities include: 

1. Coordination and collab6ration with Other agencies 
funding Development activities ; NIE is the lead' 
agency for Research and Development in the education 
sector, and is the only existing agency with ^ legis- 
lative mandate "to build an effective R5D sysjtem" 

! 

for education. Thus, it is the only organization 
that can legitimately claim some "right" to function 
in a system orchestration role. 

2. Conferences and seminars : Wc have noted several 
kinds of communication linkages which are weak or 
missing in the educationa' R/D^l system. Conferences 
"and seminars provide one way to provide some communi- 
cation between groups and to develop the imormal 
communication networks which tond to exist in more 
mature R/DU systems. Such v.ont'orcncos might bring 
together: Users and Developers (to impi.^vo the Need 
'Identification process, or to provide Dinelopors with 
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linkages to^exemplary practices in the^tield); 
-M* regional groups of Users (with a focus 6tt cither 

some specific topic of relevance to the region or 
simply beginning the dev<)lopment of informal region- 
al communication linkages); Developers (again, to 
help' Developers become more aiware of possible proj 
linkages among different Developers, and to begin 
development of informal networks); etc. 

Considerations Internal to NIE 
A. NIE Staffing and Development 

The educational issues and pottntia] NIE roles which we have 
been discussirg have implications for the kinds of staff NIC needs, 
the kinds of skills its stuff must nave, and the kinds of structure;^ 
which are or are not relevant to NIE. 

Whi^ the scope of tMs study does not permit full dcline;ition 
/ of these implications, some observations may nonetheless be made. 

1, The emphases of this analysis on: a) orchestration as a 
major role of NIE and b) process facilitation as a basic 
management mode call for organizations! structure- and 
management r.tyles and skills other than those normally 
associated with a bureaucratic type of organ illation. 

2. Further, these same emphascj in the context of the im- 
maturity of the educational R/D^il system, call for staff 
personnel in NfE wiio ore educational professionals and wKi^ 
continue to be actively involved in various aspects of \ 
the field. \ 

5. A somewhat decentralized rather than highly centralized I 
type of on ni rational structure would socm to be bettor / 
suited in gaining entry into the User system, tn develop--/ 
ing iinkngos with and bets .'cn virions parts of the cdu- ' 
jcntional system, and to fuci 1 itat ing system 'mi idiuj- within 
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the field, 

B, Nin^s View of I ts Relation to the Fiel d ( 

/ p 

Most of this analysis has focused upon an orchestration role 
for NIK. The question at this point would seem to be: Does NIE 
accept that role' for itself? Or does NIE prefer to play a 
different role? 

If NIE is willing to accept the orchestration role, then the 
Institute must begin the process of developing the kinds of goals, 
priorities, policies, structures and staffing th^t are relevant 
to and/or required by this role. In this p>*ocess, it may very 
well be that a certain amount of internal re-orientation and re- 
training would be wise and/rr necessary. We shall r.eturn to these 
considerations later ip our analysis. 
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OISSHNUNATION 

I. :HE GENHRIC C iiAR ACTERISTICS OF DISSI :MIN^'"I0N 

1. Introduction 

2. The Place of Dissemination in the R/l)ii I System 

A. System-Creating Phenomenon 

B. A Linkage Process 

C. Optimizing Interaction and Pit 

D. Many Modes and Products 

1. Many Products 

2. Many Modes of Dissemination 

3. User Familiarity and Acceptance 

4. Fail -Safe 

F,. R/DQI System Maturity 

1. Immature R/D^I System 

2. Mature R/D§I System 

F. The Role of Intermediary Organizations 

G. Long Term Objective: Developing a Mature 
System 

5. Key Building Blocks 

A. Elements External to the User 

1. Information from Knowledge Sources 

2. Quality Control 

3. Sorting and Retrieval 

4. Tailoring 

B. Bleraents PrimariTy in the Domain of the 
User 

1. Trial 

2. Adoptic^n and Implcmcnti^t ion 

3. Utilizjltion 

C. Elements either User or Producer Initiated 

1 . User Awareness 

2. Matching to User Needs 
I). Linking Producers and Users 

1. Dimensions (Sub-Blocks) of Producer- User 

Building Blocks 

2. Devising Producer-User Strategics 

h. Identifying Key Issues for Dissemination Pro .ess 
4. Factors Affecting Dissemination as a Process 

A. Cost Aiffoctivcncj^s as a Factor 

B. Professionalism and Reliability Factors 

C. Motivation and Coi-nitment 

D. Trust 

H. Roplicabi I ity of Dovo loped Di ssomi nat ion 
Mud i a 

F. Overall: A Complex Proct^ss 
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Key Prof) I em Areas 

A. Scope of the Prohleni 

1. Si2e of the User Population 

2. User (Market) Variations 
Producer/User Breakdowns 

1. Information Flow 

2. Producer Motivation 

3. User Motivation 

4. Skills 

5. Supplemental Resources 

6. Patterns of Adoption 

C. Knowledge About the Dissemination Process 
Design Requirements and Strategies 

A. Congruency and Adaptability 

B. Stability 

C. Management Mode 

D. Quality Control 

The Producer/User Match 

F. Buying and Renting Services 

G. Product Championship 

H. Stimulating User Demand 

I. The Devclopmental/Emergent/Expcr imental 
Strategy 

J* Fail -Safe Mechanisms 



Level of Maturity of Educational R/Dfi I as 
a Whole 
User Setting 

A. Scale 

1. An Enormous; Number of Usrrs 

2, A Tremendous Diversity and Variety 
Among Users 

B. Motivation and Tochni c\il Capability 
CI imate 

'Numerous but Fragmented Dissemination lifforts 
Recent- Federal and State Initiatives in 
Di s^t^minat ion 

A. Three Modes of Hi sscminat ion 

K Creation of a Comprolicns i vc Resourct^ 
Base 

2 . Product Advocacy 

3. Cluingo Process Advocacy 

p.. vSome S}HHMt'ic Current Stratcjn' Developments 

1. Cent rn 1 i r.ed Resource Basn: The l-xpansion 
of ERIC 

2. The States as hit (»rmed iate Ui sseini n^i t ion 
Median i *>Tns | 
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5. More Active Dissemination/Utilization 

Strategies 
4. Two-Way Communication 
C. A Note of Caul Ion 
t>. The Personnel Base 
7. The Institutional Base 

III. IMPLICATIONS FOR AG ENCY BEHAVIOR 

1. Procurement Behavior 

/ ' A. Orchestration as the Main NIE Role 

B. System Facilitation and System Building 

C. Process Mode of Management Control 

D. {Attention to Linkages/Transforms 
H. 'Use of Mi- M Strategies 
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G. Monitoring 

H. Mandatable reatures 
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DISS HMI N ATION 

n:i: gbx bric cha r^acturistics ov disshminatjqn 

1 . Introauction 



In the sense of a segmented conceptual model of a total R/Dgl 
process. Dissemination may bj considered as a sepaiuitc, distinct 
part of the process-^- specif ical ly as the link which takes a developed 
product* and sends or gives information about the product (or 
distributes and delivers the product) to the User population.** Thu^i, 
on this conceptual basis, we picture as distinct a separate Dis- 
semination ojement of the R/D5I process---an element having tangential 
boundaries with Producers/Developers on tr.e one hand and with Users 
on the other. 

In a more c* mplete systems sense, however, Uisscmination cannot 
be vie.vvu as a basically isolated segment. Precisely because it is^ 
t!;o link between P •.•search/Development on the one hand and User 
Sclection/Implementation/Utilization on the other, we must conclude 
that for both theoretical and practical purposes. Dissemination 
either does not exist or exists dysfunctional ly when the Research/ 
Development and/or the Selcction/Implementation/Ut ilization fi 4 ions 
occur Inadequately or inappropriately, or (in the case of Selection/ 
Iu,/lcmentation/UtiliEation) do not occur at alK 



*As noted in earlier cliapters, for simplicity of usage we us3 the term 
"produrts' whenever we arc referring to the outputs of the Development 
process. Ilnvever, the reader should keep in mind that the tor.i 
'products'' is meant hero to convey the fi^jl array of Development 
outputs-«programs, processes, models, strategies, appro. tchos, etc. 
lis well as the narrov;or range of outputs we typically think of as 
''prouuc t s • " 

**We rccognizt* that the definition of Dissemination tliat h;is achieveu 
some doj;roc of conscMisus in ocUication today includes ^wo-way commua* • 
latitn between Producers and Users, with Users *MV(^ding forward'* to 
Producers information about th(*ir perceived needs as well as evaluative 
fecu, lOk on the pr()ducts tliey have tried to uso. However, althcugh wo 
arc- I arc that a combined Dissemination and .I-eed -l"orward (D/i'lO system 

i:-jiag designed by NK:, we view l-c(*d-l'orwa rd as a concc[)ti;a I b discrete 
ftr^ tion, and tliorefote do not treat it her(^ in tins analysis. Peed- 
i.-r.%'irJ is an additiorwM fmu'tion we mi{; * cofjsidor in sokk* future, ex- 
p:r.J;*J analysis. 
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In worU: 

a. The kind of rnsscminatiofi system nc»:ded in a given sector 
is dependent on the nature and level oF jnaturitv of the 
total R/D51 system. 

b. A Dissemination system is useless (and even dysfunctional) 
if it is not designed in accordance with the realities of 
the existing overall R/DJII system,^ 

By way of contrast, it is more possible think of Research 
and Development as separate, isolated elements simply because there 
is an identifiable e;id result— there is a -"product," even if it 
figu/atively "sits on the shelf" and is never Used. 

Similarly, it is also more possj-ble to think of Implementation/ 
Utilization as separate, isolated elements. Users do implement and 
use products, methods, processes, etc. -even though few Users may 
1:3c a partic'^iur prcJuct; or the product is of poor quality; or the 
product is used inappropriately or inadequately. Also, these products 
may or nay not have come from external RSH-bascd sources (in a 
direct sense). On the other hand, the "end product" of Dissemination 
is the effective Selection/Implenientat ion/Utilization by relevant 
Users of products which are both adeqiM tc and relevan t to the User's 
need. 

Thus, it makes sense to think of the Devclopment/Disser ' nation/ 
Implementation/Uti lii:ation process as a pa day: o with the focus bcin^ 
on the User. 

To illustrate, where the overall R/I?r.I system is immature, several 
conditions will tend to exist wi.ich si gni f ica'ntl y irpact upon the 
effectiveness («:nt. thus upon the dcsljjnj or the Dissemination system. 
1. The transforms between the steps of the R/DiW process 
arc not clear. Tl»at is, it is not ol)vious (for e^cimple) 
how Kc.searcli results can bo applied 'o Dcvclop!:!oh|:; how 
qu.'-.lit/ control is to be done in rclntion to Prod'uci ion; 
h •>..■ th'.- p''")duct will •.»ot to ihc User. 
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2, User knowledge tends to be highly limited. The User 
may not know a product exist .-, or how to differentiate 
between a *'good'^ and a ••bad" product; or how to effec- 
tivcly use a prodm t — or, because of these factors, 
the User inay not even try the product. 
Thus, when the R/D61 system is immature, certain kinds of Dissemination 
policies, strategies and mechanisms may be needed which are not needed 
when the R/D5I system is mature. 

Viewing the Devclopment/Dissemination/ImplQmentation/Ut ilization 
process as a "package, it now becomes possible to consider the 
question of appropriate Federal agency policies and strategics in 
Dissemination. As a preliminary comment, we may note that Agency 
options are limited by certain factors: 

1. The further the Dissemination "package" goes into the 
User system, the 16ss control can or should be maintained 
by the Agency. That is, the Agency may have high control 
over the Dissemination aspect, less over User Implc-- 
mentation, and even less over Utilization. 

2, The larger and more fragmented the User system, the more 
difficult it becomes for the Ag •cy to be directly involved 
(in an effective manner) with Users. 

The discussion which ^ollows is predicated upon the above under- 
standings of the Dissemination process and of Dissemination s/steias. 
'./o will first examine what role Dissemination has in the overall 
:t/Dfil system. From this context, we will }>e able to identify some of 
tlic key building blocks of a Dissemination system, Kv\t , wc will 
focus on strategic elements of Dissemination as a process, and then 
on some of the likely key prol)lem areas ior Oissemi u. t io . , '-c will 
.lis.) comiiUMil on the styito of knowliHl«<* copce^ininj; Disset... nation, as a 
total, complex process. ^ 

Fror.! the above* we will su^;Rost some relevant Dissemination 
strat- tj.^^. The ontlino and process of the discussion uliich follows 
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may be represented schematically as in Figure 1 

FICURU 1 



The Place of Dissemination 
fin the R/Dtil System 




'>>Kcy Building Blocks 

Dissemination as a ProcessT^ '-Ov Design ^Strategics 

Some Elements — Requirements 

pKey Problem Areas y/' 

Knowledge about Disserama- 
l>tion as a total, complex'^ 
process 



The P 1 acc of Dissoftiination in the R/D^i Syst em 



To ^..i.rstand the nature of (and effectively develop) a 
iii.sse.ninarion system, it is vital to have a clear understanding of 
how Dissemination fits into and affects the overall R/D^T proccs 

t^^ _Syste m-Croating Phenomcjion 

In the first place, we must be aware that Dissemination is- a 
system-creating phenomenon. 

This is true because without Dissemination a total R/Df,! system 
simply does not really exist. Research and/or Devolo{-nent can 
indeed exist indepoTvlently--but unless there is also a link to the 
Users. Rcsoarch merely produces knowledge in isolation, and Devnlopmcnt 
merely produces products which, in effect, "sit on the shelf." Thus, 
mechanisms for dissowinat i ng the results of Research and Development 
to Users must exist for a total R/D{il system to exist. 

Where the R/Df,! system is immature. Dissemination mechanisms may 
have to be cre.it ed (where none exist) and/or supplemented and sup- 
ported fwherr inadequate meclianismr> exist). Where the R/DM system 
is mature and formi 1 i .hm . i nst i tut iona! i ".ed l))sseminat ion 
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arrangements may exist in the form of mechanisms that permit direct 
PrrJucer (or Svipplicr)/User relations. 

B.\ Linkag e Process 

It follows frora the above that Dissemination, in its essence, 
is a linkage proces$--a linkage process so important and so critical 
that the total R/Diil system does not exist in its absence. The 
functions of this linkage process are essentially thrce-fold: 

a. To inform Users of the results of Research and Development 
which are relevant to and usable by them. 

b. To enable Users to effectively utilize the results of 
Research and Development. 

c. To enable Developers and Producers to develop and produce 
products which fit User needs. 

Viewed from this perspective, several important implications become 

a. It is vital that the Dissemination systems operate with 
clear understandings of the User organisations with which 
they interact . 

b. Dissemination systems and mechanisms must be tailpred to 
the nature and style of the User organizations. \ 

c. The products to be disscmlnateJ must he matched both to 
User needs , to User capabil i tics, and to User read iness 
to adopt new products, programs, processes, even new 
D i s sem i nn t i on mech a n i sms . 

d. It is important to keep in mind that tho rolovance of any 
specific Dissemination mechanism is dctormined by whether 
or not it helps achieve Dissemination purposes. A specific 
Dissemination mechanism is not important in and of itself. 
Thus (as wo shall Inter notol, in any yjvcn situation, 
several dirfcrcnt Dissemination niechan i sms may be reasonaliiy 
effect i vc . 
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llo..ovor, we must also note that normally most Users will tend to be 
to adapt to new information sources, new mcchanUnis, new modes 
of operation, new materials and products. In this regard, it may 
at ti~?s be good strategy to utilize and improve an existing Dis- 
semination mechanism (with which the User is familiar and comfortable) 
rather than to introduce new Dissemination mechanisms (which in them- 
selves may require change by tlie User). 

C. Optimizing Intera ct ion and Pit 

Finally, from all of the al»ovc, it follows that in ti\e. design 
of Dissemination processes there is a very high premium on desi;-ining 
Dissemination mechanisms that will optimize interaction with I'licrs 
and will thus also tend to optimize the fit between Products and 
Users. However, as a strong caution, we must also make a distinction 
between c reating mechanisms for Dissemination and achieving Disscmina- 
ti- vct ivcs. That is t9 say, the mere existence 'of Dissemination 
.r.c/hmi.^ns Joes not guarantee that Dissemination will actually occur. 

D_. Ma ny Products andJjodes_ 

From the previous discussion of the importance of dosij'.ning 
Dissemination mechanisms which optimize interaction with the User 
and fit between Product and User, an erroneous conclusion could 
easily be drawn--!. e., that there Is one best set of Dlsscmiiuit ion 
mechanisms and that Nil; should discover and design that set. There 
arc at least 'four important rcisons why such a conclusion woiild nol 
only be erroneous, but also potentially dyslunct ional . 

1_. Maiiy Products 

How a Dissemination systci.i will fimctinn in pnictioo 
depends to a significant degree on the types of producis 
being disseminated. Hvon within a given soctor {e.g., edu 
cation or health), there will be a degree' of prodiut variation 
•sufficient to call for certain differences in Di ssani n:it ion 
nochanisms. For exampl( , the method'? most appropriate for 
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Dissemination of a very simple product arc likely not to be 
at all appropriate for the Dissemination of u very, complex 
technology; and vice versa. Thus, th^c must, be a fit between 
the *Uvhat" (Product) and the *Ulow*' (Dissemination) --a fit 
that will tend to vary across product types. 

We may also* note here that this linking of product type 
with types of Dissemination mechanisms can, over time, lead 
to some degree of specialization by Disseminators in relation 
both to the type of products ^hich they will tend to dissemin- 
ate and the Dissemination mechanisms which they will tond to 
utilize. 

2. Many Modes of Dissemination 

In aj^i^ given situation, there are likely to be several 
modes or mechanisms which would be reasonably and more or less 
equally effective for the dissemination of a particular product 
or tyre of product. In such a sitiuition, the issue is not: 
IVhich is the best? ; but rather: Which is available? V/hich 
already has User confidence (or lack of confidence)? What 
ni*e the cost differentials? etc. 
3^ Us er Familiarity and Acceptance 

It is important that Users have sufficient confidence in 
the source of Dissemination information so that they will be 
at least initially favorably inclined to try out an innovation. 
It is not unreasonable to anticipate that in many instances, 
out of all the possible relevant and roasonable Dissemination 
mochanisas, a U'^er (or group of users) will have confidence 
in one particular Dissemination mechanisms because of pi ior 
positive cxp^nnencos with it. Thus, where possible, it would 
bo roasonable to utili/c that mechatrism. Given the largo number 
of Users nationwide, wc would expect different User sets to have 
Ji^Terent '^favorito'* Dissemination inccliani sms • Pach separate 
I-i > semi n;:t ion nochanism may gravitate towards a specialization 
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in the uissemination of a relatively limited line or type 
of product, but for that line or t>'pc of product, it is 
potentially a highly effective Dissemination tool for a 
given set of Users. This need not be \ lowed as a factor 
limiting improvement in practice. As we have noted, a wide 
variety of different innovations may be potentially helpful 
to any given User; and presenting a User with tpo wide of a 
set of alternatives may result in a dysfunctional state of 
confusion. 
4. Fail Safe 

As will note more fully later, the Dissemination system 
needs to be made fail-safe, since it is reasonable to expect 
that Users will. from time to time have "bad" experiences with 
Dissemination sources, which when going through a "learning 
curvc'^ .T-ay lead to premature and long term rejection. 

Ue have just noted that when. the User has had "good" 
experiences with a Dissemination source, the positive experience 
should be reinforced and capitalized upon. The reverse is true 
when the User has had "bad" experiences with a Dissemination 
source. In this situation, reinforcement should be avoided— 
but if there in only one Dissemination source available to the 
User, the "bad" experience will be reinforced and the User 
will tend to discredit and reject the entire Dissemination 
system. 

Thus, it makes sense to provide the Users with several 
al ternativ e Dissemination sources, such tiiat if they experience 
failure with one source, they can go to another source instead 
of rejecting the entire ULssominat ton system. 
In suamury, there are mnny Dissemination modes or mechanisms 
which could pot.!ni.:lly optimize interaction with the User and the 
fit between prod', rs and Users. No sin^U- mode or mc:hani si:;s (or 
even set of r.cch..r. I <n.s) is Hkclv to ho clearly "Iho host." Further, 
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there are a number of very good reasons to follow a policy of per- 
riltting and supporting the successful functioning of a variety of 
Dissemination systems. 

£. R/DSI System Maturity 

The type of Dissemination system required varies as a function 
of the degree of the maturity of the R/Dfil system., 
i; Iimiuiture R/Dgl Systci" 

When the R/D§I syster is immature, several key conditions 
will tend to exist: 

a) There will be general lack of quality and quality control 
in Development (and also in Research) , 

b) Users will generally have a low level of effectiveness in 
Implementation; Users may be (to a large extent) unaware 
of what is ava.Uablo for their u?e; and Users will lack 
the evaluative capability to distinguish between "good" 
and "bad," "relevant" and "inappropriate" products. 
Under these conditions, the Dissemination system must be 

designed to find, evaluate, categorize, store, and retrieve 
information about what is available. Further, the Disseminatioiv 
system must provide mechanisms to onj^ble Users to be aware of, 
properly select, and be capable of implementing the partiq^ular 
Development product which is most relevant to and usable by the*. 
Such mechanisms may be provided either by the creation of new 
mechanisms or by the utilization, modification or improvement of 
existing mechanisms. 

Muturo R/ Dftl Sy stem 

When the R/Urj system Is more mature, a different set o*" 
conditions tends to exist. 

a) Produt ts of good v. mlity and mcchnnisms for quality control 

will be hiore readily available, 
h) Users will tend *o hnvo a hip.hor degree of familiarity with 

products; they will bet I or know liow to Find, select, i;nplc- 

mcnt, and use ,unv products. 
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Under these conditions., an agency wUh an R/Dr,I system-build- 
ing niis«ion Kill 'iavc less need to be poncornod about estab- 
lishing Disse:nir.ation raechani ..ms (though there may still bo such 
a need) and Kill be able to focus its attention on different 
specific aspects of the overall Dissemination process (e.g.. 
ensuring outreach and disseminatins thin market products to small 
special groups) . - - 

V. The Ito'lc of Int-ern iediary Organizatioiis 

The role of intermediary Disseminntioy. organizations deserver, 
special attention at this point. (By intermediaries wo r„ean organi.,a- ,, 
tions external to both Producers and Users.) 

As described above', the conditions existing when the R/Df,I system 
is iimnature may require that intermediary Dissemination organizations 
rnd mechanisms exist to perform those functions which are net bexng 
— -ora-d (or are belr.- inadequately performed) by Producers/ 
Developers and Users (e.g., quality control. User search, testing 
evaluation). Indeed, it is likely that the Agency will have to cstab- 
lish such intermediary mechanisms. 

However, as the R/D6I system matures, the need, for intermediaries 
may decrease ovc. time and the roie of intermediaries will change. 
For example, as Producers be ome more capable of developing quality 
products, the need for intermedia quality control mechanisms diminrshes 
-Intfe^. the need for some intermediary mechanisms and functions r„ay 



completely cease to exist 

A key issue arises for the Agency in connection with these mter- 
mediarv mechanisms. Organisations and institutions, once established, 
tend to seek to continue their existence. Thus it becomes imperative, 
from the beginm..^. that tl>e Agency build adactabiUlJ'. 1"^° l^l'^ 
intermediary mechanisms it stimulates and funds-and even provide for 
the possible demise of these mechanisms. 

A^basic long-term objective of the Agency should be to facUitute 
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the transition' of an immature R/Dfil S)';:tojii into u more mature syytem. 
"Thv Dissemination syscem sholild he seen as a key clement in achieving 
thii objective for two reasons: , . 

a. The vital role of Dissemination as a linkage process 
requires that mature Dissemination mcchanismsi bo a part 

of any mature VD§I system. : 

b. Under the conditions existing in ah immature R/Dfil system, 
the intermediary Dissemination mechanisms tend to fulfil 
functions usually associated with Producers and Users in 
more mature systems." Thus, Dissemination becomes a 
potentially strong entry point for facilitating the maturri- 
tion of other par^ts of the total R/D8I system. 

3. Key B uilding ^Blocks of a Dissem ination System 

From an understanding of the overall R/" fl system and of the place 
. of L-i55er.rlnation in that system, it now becomes possible to identify 
so:;ic :-Joy elements (or building blocks) of a Dissemination system. 

A. Ele ment s whi ch are Pr iiya£Ujr_Jxter i^^^ 

^ ' InfornKition fr^om Knowledge Source^ 

Mechanisms must be provided that can determine what is 
available. This could be information about products 
currently under development; products that have been around 
for a long time but liave not been widely ur^ed; or oven 
exemplary practices witliin User orj',anizations which have 
not become known outside of a particular User organization. 
2 • Q uality Con tro^l 

Mechanisms mu.U be provided to sort out tlic "good" from 
the "bad" products, i.e., those products whicli are useful 
and relevant as contrasted to those products which do not 
work or are not appropriate. 
3 . So2;Ujii^indJRjM:H^^ 

As there will likely bo more products than a User can , 
reasonably be expected to be awnro of, some mechanism must 
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■be 'developed-iihich viir.'anow^^ 

■^he particular products which arc relevant . to their neods. . . v..^^^ 
4. Ta- lorir.g 

-Because of the diversity among User organizations and their 

needs, it is probable, that .existing products will not quite -fit" 
User needs and will thus have to be "tailored- to fit a par, 

ticular User organization. 

The relative definitivene.s of the problem/solution fit in 
a given instance would significantly influence how these differ^- 
ent elements might be used in working with a given User.; In 
some instances, -there might be a variety of innovations that 
could assistUsers in their (often loosely defined) needs, and 
one mi^bt decide to provide them with information on a broad 
array of alternatives.' In other cases, however, it might be , 
far fro.i clear which products would be more or less useful. In 
these circumstances, it becomes the objective of the process to 
eliminate the clearly inappropriate, while narrowing down the 
choice to roughly comparable alternatives from which the User 
could select. 

B^li!E^nH_!^?liH^ 

in "the initial stages of Dissemination, whether initiated 
by the User, by Producers/Developers, or by an intermediary 
agency, the nature and mctiiod of Dissemination must be 
designed so as to motivate the User to give the ne. produce 
a trial. Unless this is done, the overall Dissemination 
process breaks down and stops. 
2 Adopt ion and Implementation 

Dissemination is not merely a "sanding out" of information. 
Dissemination must also be concerned with what is done with 
the information after it is received and tried by the User. 
This should be obvious, for the purpose of "spending out" 
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■ . informaU au. 'is ;t^^^ inc-ixviso the, Adoption and Jiuplcnuont^ • ; 
; ^ .;o£ new..^ materia is ^ , pro cesses, vet c ::Thc point: is. -v.^ 
■ \ tv«:iphasi7.cd here bocaui'>e Disseiiiination often is understood 
simply as *^sending out/^' with a resulting neglect of \-hat 
'Users do — or are, able to do--with the information they 
• receive.'^ i v^ '':" '^^ . y. -^- -a- ■ ; .vrr ..v - 

. 3 » Utilization 

Dissemination systems must also include provision .for 
continued Utilization — not just Adoption and Implementation 

followed by discontinuance. 

. . ■ . * 

The role of Dissemination in Adaption, Implementation, and 
Utilization is to insure that Users will receive the help they 
need to make effective use of what they receive. ^ 

C. Klemen ts which ma y be Fither Producer or User Init iated . 

1 . User Aw aren ess ^ 

Mechanisms must exist through which the User can become 
aware that a product exists which may be rolev^uit to his 
needs . 

^ ' M^t:ching to User Needs 

Mechanisms must exist by which the sources f.nd/or Sup- 
plier of products and information on products can become 
aware of User needs in order to permit t]\o sorting/ retrieval 
and tailoring described above. 

In both of the above, the initiative may come validly from 
the Users, from Producers/Devolcpers, or from intermediary 
agencies, 'Hv^ Users may be looking for a particular innovation, 
or a message may btj sent to the Users to inform them of a par- 
ticular innovation. 

> 

il^^ ^ ^> ^ng Producer/Use r Linkaj^.e Str^^ 

A> nime^sums^^j^^ Building Blocks 

tiach of tho building blocks considered above might be described 



•yfck r-lnf^ kirovacdgc sources^' nii.ghta)o elaborated.:;^^ 

'i^y reference to such 's^^^ printod 
ni'ateriali>, face- to- face presentations, conferences, demonstration 

i'pro.grams, etc ' V • >■ ' ' . ■. '. 

^ The- aesign of .Dissemnat ion strategies essentially a 

; process of combining these building blocks, and especially tho 
various building block dimensions or "sub-blocks". The cffcc^^ 
tivenesti of these strategies is likely to be dependent on the 

^xxtent to which these component building blocks and "sub-blocks- 

■are properly selected, sequenced and timed for a- particular 
target User and a particular innovation. ' For example, in one 
situation an appropriate Dissemination strat.^gy might consist of: 
a. series of introductory "flyers" or brochures; a visit to a 
User to ascertain needs; search -and sorting to identify apprp- 

■priate available products; a second visit by a representative of 
the disseminating agency; perhaps followed by Informal discussion 
to zero in on a given innovation; a demonstration by other curren! 
Users of the innovation; and a promise of needed service and 
support. In a different situation, such a Dissemination strategy 
might be inappropriate. 

Prom the above we can identify at least three key issues 
that must be considered in the design of Dissemination strategic. 
1. What is the appropriate, combination of "building blocks" in 
a given situi'tion? 

•2 What in the appropriate organization for implementing the 
Dissemination strategy in a particular situation? In determinrng 
the appropriate organization, some relevant questions might be: 
Who works with the User? Who has the relevant knowledge and 
sl;ill3? V.no knows (or can obtain sufficient knowledge about) 
the particular innovation? 
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.5..,. ."^Vhat type of role Is , tlu^ Agency Ip hayo iu iiupUMucnting a 

■ i:;Pi cm inat ^ 

■•^^ in the Dissemination process, or wi-l.l it 

■ play an indiroct role facilitating the activities of Producers, 
Intermediaries, and Users? In cither case, what will bo its 
major objectives and modes of operation? ::.-:-y::--.v--::..:--^r.-rr. 

5. Factors Affecting Dissemination as a P roce ss; . 

A. Cost/ Ef fectiveness as a Factor 

At the very outset, one must be aware that Dissemination is a 
very cost-dependent process. The Disseminator has large publics at 
both ends of the process. That is to say, the Disseminator must be 
in touch with the various Producers and Developers, on the one hand, 
and with the various Users, on the other. Further, there are a variety 
of Dissemination media to be considered, each with different costs 
attached. 

Much of the discussion about Dissemination in the literature 
focuses on the effectiveness factor, but less emphasis is given to 
cost/effectiveness. This is unfortunate, for the Disseminator must 
consider cost/effectiveness of various Dissemination media in relation 
to the dissemination of a particular innovation to a particular User 
public. Fur example, mailing 10,000 brochures may result in a low 
percentage of response (in terms of Adoption, Implementation, Utiliza- 
tion), but such a mailing might be relatively inexpensive and could 
reach a very sizeable audience. Hence, even with a low response rate, 
a substantial impact might be attainable for the cost and time involved. 
By way of contrast, having three staff members make trips to User sites 
could tend to be more effective (with the Users contacted) but might 
be relatively more costly and time-consuming and would roach far fewer 
Users. Thus,, cost/effectiveness considerate us would be a key factor 
in. choosing between these two illustrative strategies. 

ViC must further note that in most instances, it will not be 
obvious (or even determinable in advanced) what strategy is in fact 



"«rong «ays" of Dissemination relative- to a particular innovation and 
^Sa;;patticular ys« pubUt. and identifying; t^^c.o, "m<>^^& «ays" M:^yMM, , 
:>' importOTt issue for the: Diss^^^^ 

Most likely, an optimal strategy will be to include a varictjr ■ 

of approaches; and the specific co,»bination of approaches that should 

be used will bo dotern,lned by such factors as: objectives; nu..ber and . 

nature of Users; cost; availability of Dissemination resources (e-g-, 
- materials, skilled personnel . support, equipment such as audio-vrsual 

"equipment, etc.); etc. 

In addition, as we have said, there may be much uncertainty as 
to. the benefits derivable from alternative products. Since products 
r-. vary in cost to th. User, and there is often also an intoractron 
■ -between the nature of the product and how (and hence at what cost) ^ 
■ it can be disseminated, there can be both considerable variation and 
considerable uncertainty associated with overall cost/effectiveness 
: considerations (where cost is now seen in both User and Dissom.nator 
terms) . * 
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■ Whatever approaches the Disseminator decides to use, it is vital 
that these be reliable a.,d be used in a professional manner. Sloppy 
brochures, publications th.xt are not delivered when promised, products 
and programs that are inappropriate, unclear or .nusablc-when 
unprofessional and unreliable Dissemination offor's such as these occur, 
the i^cdiato and lon,-term effects can be disastrous for the overall 
msseminaticn process. In the short term, the result is UKely to be 
that the User will give little or no attention to the innovation. In 
the lon» terr.,, the result is likely to be that the User will reject 
out-of-hand any future Dissemination efforts by the Disscninator- 
whutever the Dissemination strategy. 
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The basic ,a>niso.n,.,fo^^^ is , t:hut: tlio {Ji^crr 1 iyos iii an 

thD - User cannot afford tho luxu.tv)< - nf an .itcn\it:ivo p 
■ ment/i.7.prove/oxpor;lnient/imp.rov 

' C. Motivat ion and CoiT im:i ti«t3nt 

From the above, it is obvious that the Dissemlriation prucoss 
requires a great deal of hard work. Further, the proces? is subject^ ^ ;'. ^ 
to a great deal of potential discouragement on the part of che Dis- 
iseminator. Hence, Dissemination need:; to be done by persons and 
organizations who feel, a; degree of . "excitement" . about their . tai>Kr-i .e. ,.. 
those who are interested, in and coinmitted to the particular innovation . 
and/or to improvemont in the sector as a whole. 

p.. Trust 

An effective Dissemination, process is based on trust- -trust in . .. 
the infoination being disseminated, and trust in the source of Dis- . 
semination. Quality control is a key to building and maintaining 
trust'-quality control on the products and quality control on tne 
usage of the products. Simply put, the Disseminator should not be 
promoting a product under conditions that will lead to its misuse 
(e.g., under conditions where the User docs not know how to use it; 
or where the product is inappropriate to the User's need). 

We may further note that interpersonal mechanisms tend to be seen 
as more trustworthy than printed and otlier impersonal mechanisms. This 
is especially true in areas of innovation where there is a low level of 
development and a high level of uncertainty. 

}L Hcplicability of Developed pi sHejnj^^ 

When considering any particular element of a Dissemination 
strategy, it is important to make a conceptual distinction between 
the der.ij£n stage of devising the strategy and the subsciiuent op erationa l 
stage of using it to achieve the Dissemination objectives. Tiu; design 
stage is likely to take considerable time--decidlng what to do, develop- 
ing it, making it usable through pilot testing, evaluating, refining, etc. 



nicjJt a,^.the. i)uxt^i .'. 
effort. Similarly, traininR of "detail men" once the pattern has buon 
established tends also to be a relatively short-term effort. 

This ciiaractefistic of, replicability canlbe u^^^^ \' 
iexpand Dissendnatipn programs'lTa^ 

uoing, it is"' important not to follow the tendency to overuse particular 
dissemination media simply because it can bo dont> more cheaply and 

easily..,...^..;,,., , .... 

F._Qyoral l : A Complex Process 

Though at times it may appear deceptively simple. Dissemination 
is a complex process. Tliore will bo several steps in the process, 
and tliese are likely to be interactive. For example, poor quality 
control which results in a "bad product" being exposed to a User may 
negate the best Dissemination, strategies . Similarly, unprofessional 
and unreliable Dj • •nomination efforts will likely, result in reduced 
feedback from the User [relative to the innovation), thus reducing the 
User information needed by Researchers and Developers. 

Thus, while an individual clement of the Dissemination process may 
be relatively simple and manageable, the overall process is complex. 
.There are many different steps in the process. There are many different 
kinds of Users, Developers, Producers, Intermediaries, Products, etc. 

It becomes obvious, then, that designing and managing tl\e Dis- 
semination system reciuires considerable skill, anl this becomes a key 
issue for the Agency funding and building such a system. 

^ • jCey Pr oblem Are^is 

Prom an understanding of the process, it is possible to 

delineate several key generic problems which must bo dealt with by the 
li'-syeminator if Dissemination is to be effective. 

I 



V . In. aiiy iicctor/.thcra a largo number, of potoutlal Users ' 

of innovations. In the cducatio'- sector in particular, there 
may be potentially thousands o£ Users nationwide. Just making 
effective, contact with so many^ U likely to. be a signifi- 

vcant :;prpbl-em :in. itself .:;..^ :;f ^;;:;v; v/'^.'.:; .y::::::--:.:':^'--:^: --^^ 

2- Uscy (Mar ket) Variations 

Within any largo User population, it is inevitable that there 
wil,l:bc yariations among Usors. There may even be variations 
across types of User sub-groups . Further, variations may exist 
across different facets of User populations and sub-groups. For v 
example, Users may vary according tot needs; location; size; 
private vs. public status; profit vs. non-profit status; capa- 
bility to acquire, \test, evaluate, implement and utilize innova- , 
tions; etc, In the education sector in particular, the variations 
are likely to be qi^ite numerous. Given such variations, several 
specific problems ijiay arise. For example: 

a. It may bo difficult to identify which Users are the relevant 
potential target population for Dissemination of a particular 
innovation. 

b. The needs of the various Users are often hard to identify and 
define. 

c. Market segmentation may be necessary. 

tlucer/ Us er_ B reakdo wns 

A second key area for potential problems in Dissemination is tlie 
relationship between Producers and^ Users. We can specify several 
conditions under which the direct Producer/User relationship is likely 
to break down i 

^* Infarmation Flow 

Obviously, the flow of information betvtfe.in Producers and Usors 

is a key element in the Dissemination i)rocess. However, this 




•a... . -ThcTO. -are , laiigiiagc ^<Ui'i\-r(;n/'ei> bet.\.*iJon l'roduc<vr;i arid Usurs 
• (e.g., the use of R(il) vs. operational laiigua^o and v.oncept.s) . 
■ For another example, in certain education situation^, key 
!.-^^^:^^psers^m^^^ tfie Pi-oducors aro 

. b.': There is. a lack of adequate cenununicotion channeiy between 
Producers and Users. This is er»i,cc;ially likely to be the 
.. ... ..case, ivhcn the, R/D^I system is inunature or where the market 

'c. .Users lack the caphbility to understand the technical 

aspects of the innovation. Hero, even when the information 
disseminated is technically adequate and reaches the relevant 
Users,, they may be incapable of using the infomiation received. 

2 ' ■££?4H.'^ Motiva tion 

For a variety o.t reasons. Producers may have a low level of 

motivation to disseminate new products, especially in the face 

of resistance, uncertainty and/or fragmented markets. 
• a. Sonie Producers may think of themselves s imply as Deve lopmen t 
organizations and may have little interest in carrying out 
the Dissemination function. In these instances, their 
organii'-ations may have a low level of Dissemination experience 
and skill, a fact which will also tend to lessen the motiva- 
tion to disseminate. 

b. Under conditions of perceived resistance, uncertainty or 
fragmented markets, even those I'joducer organizations v.hich 
aro oriented towards Dissemination and which therefore have 
the necessary Dissemination skills may decide that the cost 
and risk are too high to attempt u high level of Dissenination 
_j An exjuiiple would be school textbook publishers wl\o r.iight bo 

- very hesitant to produce and promote innovative materials 

Khicri have not yet been "accepted" by Users--or wliich perhaps 

ERIC /^^'^ 



" ■ • . I\)r' cffccfi vu Oi-'iseminat ioii to ocl'Uv bctwtH)(i Prodiu't-r;. and 

Uocrii, the Uscriv must be wot. i vatod to i-ucuivf, tost, try and 
••7,/-,a:--v^^^^ ^ ■■■ cvaluato ■■innovat ions from Ih'oducorsy--' A ruiinlH^r of' ■i\ncto]\s - iici: --"-^ 

■;^'V;;\-;\' :V' :\ ,,; ^«?. ■potential '^'b^^^^ receptivity . 

■: .-. a. Being exposed to an innovative product can raise thv 

po;ssibility (to IJserij) that they arc not eurrently 
"doing it right," or at least not doing something as weJ.l 
' / as it m)ght bo done. This is not always an eas>y thinv, for 

; V ■..a.-..User.. to ..admit;. 

b. To study, test, evaluate and then implement and utilize an 
innovation takes time and is therefore an interruption in 
the^ work of Users who may already fool overwhelmed by deriUiiids 

' ■ • on their time and enerp.y. Where training is required for 
■ ' effective usag"', additional demands are placed on the time . 

" and energy of the User, 

c. The above discussion woi'' i seem to imply that Users generally 
' are confronted with a small number of innovations. In many 

instances, it is more likely that the User will be confronted 
wit!i a plethora of innovations; When this is the case, tlie 
User is likely to feel overwhelmed by a seemingly impossible 
^ task of reviewing, testing, evaluating, etc. --and may thuH 
be inclined to ignore everything offered. 
A somewhat different issue is Hivolved in those cases where 
an innovation lias been developed by a User. In Vhis ease the Oissemina- 
tion and the completion of the Development process 'become interwoven, 
Tlie innovation has to be identified, possibly modified and packagevl for 
a wider User group and then disseminated. A key pioblem is that there 
is generally little to motivate a User/Developer to become involved in 
this process. 
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Dissemination requires special skills, which may differ 
according to th- type of User and/or the type of Innovati.oii. . 
Thus, the Producer/Us(?r relationship may break down .whore: 
1> the Producer is inexperienced in Dissemination ; 
■■ ■ the Producer is inexperi enced in Disseminat ion in a 
particular market or in relation to a particular type 

. of innovation . 

b. User Implementation/Utilization Skills: 
; The Producer/User relationship may als.T break down when 
■'.the User lacks the skills necessary for Implementation and 

Utilization'. It is not enough to assunig th-.t a "good" 

product can be effectively used by a User. Training may be 

required. 

■ c. User Dissemination Skills: , 

'As before, where the User is also th Developer' there may 
. be a lauk of skills in knowing how to ' ^xt innovations for.' 

Dissemination, how to tailor for genera'' .applicability, and 

bow to communicate to relevant others. 

The Producer/User relationships may also break dow!i when 
effective Implementation and Utili^^ation require supplemental 
resources which the Producer does not provide and/or the User 
docs not have (and may not be, able to acquire and/or ucilizc) . 

6. Patterns of Adop tion 

The rate of User acceptance and absorption of .new information 
and new infoinnation sources tends to be slow. What is important 
to n.:te, here is that the rate of User absorption of new information 
sources-his awareness of the existence of such sources, his 
j^aerest in making use of these sources, his trust in these sources 
^-etc. --tends to be -much slower than the rate at which the.e informa- 
tion systems can be developed and made active. 
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""As^-a. .result, there is the. daiigor that creating Iliv^j,c . , 
..,.i)i j?aomination_ -systems' may load to high oxpoc tat Ions (by both 
■:VV; ^-M-thc Disseiidn the Users) , while in reality ; the v benefits A; :K 

unci s^itisfaction may be strongly limited by the slow rates of 
absorption and utilization. 

..In a word; the capacity to develop and activate Dissemination: 
mechanisms tends to be greater than User capacity to absorb and . 
utilize the information sent to them by the Dissemination mechan-' 
isms. Thus, in designing and creating Dissemination mechanisms, 
the Agency needs to assess carefully its expectations in terms 
both of the expected levels and the anticipated time-lines of 
User Acquisition, Implementation, and Utilization. This assess- 
nt also needs to be made by the personnel of theS^gwly-created 
ssemination mechanisms. 

Knowledge about the Dissem in ation Process 

The discussion thus far Has focused on various elements of the 
Dissemination process. Now it is important to note that while much may 
be known about specific elements of the Dissemination process (e.g., 
about the impact of written as compared to interpersonal communication), 
relatively little is known about Dissemination as a complex total 
process. We must further note that merely "adding up" our knowledge 
of individual element will not provide" a useful understanding of the 
Dissemination process, for these parts are interactive and influence 
and change each other and the overall process. 

Given the uncertainty associated with the workings ^| the total 
Dissemination process, there is validity in supporting processes of 
natural dcvelojiment which would lead to the use of mixed and variable 
strategies in a Dissemination system design. This contrasts with a 
strategy of seeking optimal designs at this point in time. 

^- t^^esLgn Uequ irements and Str ategi es 

Up to this point we have been developing an understanding of the 

R/Oill preces_s_. The importance in so doing is to incvuase our ability 

» > 




to design yffoctive Disjscmuiati.of. systems, develop el'feot.ivo Dissuin- 
inntion strat.oi»lus, and thereby, to reco}>nik;e tlio ;i}>proprialc ^l\oucy 
.role. It is- to this task, that we noi^ 'turn our uttentiQii. 

A. Coiij^CMicy^amd^A^^^ 

A Disscmliiatlon system must be congruent with the statu of develop- 
ment of the R/D(1I system it serves. What is needed (and effective) for 
an immature R/DSF system may not be needed (or effective) in a mature 
system. Indeed, the need for some aspects of the system may ease 
to exist. 

The Dissemination system design must be flexible and 
adaptive over time, and even provide for the termination 
of some of its parts. 
Strategy; 

Institutions tend to be slow, to change and even to 
resist change. Certainly, it is not typical of an in- 
stitution to seek its own demise. Thus, an important 
strategy for a funding agency is to build adaptability 
(and even temiination) into the imtlal design of the 
overall Dissemination system. Dissemination sub-systems 
(e.g., regional D/TF systems), and Dissemination mcchan- . 
isms. ^ 
B. Stabilit y 

Though institutionalized Dissemination mechanisms must be adaptive, 
they must also have stability in order both to avoid dysfunctional dis- 
ruptions and to benefit from the confidence and competency which comes 
from experience and familiarity. 

When change is abrupt and continual, a number of dysfunctional 

results tend to occur: 

1. Dissemination agents do not liave time to accomplish their 

objectives. 
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2. Dlssoininat.lpn agcutr. w.l.1.1 tend to bucoiiio uncertain, 

confusoJ, insocu.vc. 
5. DisHOnination agents not gain the cojif idoncc and •' 

coinpotcnc-,0 that comes from oxpcr.i.cnco and familiarity. 
<K Dissemjiuition agents will not have crcdil)ility with Users. 

5. Users will become confused and will tend to resist further 
Dissemination efforts. 

6. The Disseminator/User relationship will lack continuity. 

7. Communication channels will be inadequate and tend to be 
unknown to the User. 

Desi gn Rc quireinent ; 

The Dissemination system design must provide stability 
and gradua lnes s to cliango for the system as a whole and 
for its parts . 
Strategy: 

Stability and gradualness of change should both be a 
})avt of the initia l system design and an integral part of 
\ the funditig support and prograimning of the Agency. 

^ • Management Mode 

The nature of the control mechanisms used to manage a system 
is likely to have a significant effect on the nature and effectiveness 
of the system. Further, the appropriateness of different management 
modes depends to a large degree upon the" level or state of the system's 
maturity. 

1. The mode of management control which i«: relevant to 
the current 'level of maturity of the overall ll/D^l system 
must be built into the Dissemination system design. 

2. Provisions must be made in the Dissemii\a,tion system 
design for the management mode to oliange ov9r time as the 
ovei'all R/Dtil systCMu matures . 



3. Iiifonnatiun feodlun-k niochan.isjiui iinv-.t ))c an cv.su'nt.jal 
eleraer.t. of the U Is. semi nation system Josii;n to provi^U; 
system managers with indicators of the changinfi mnturity 
level oi" the sy.stcni, and. to signal tlio nood for clumae 
.'in the mode oi; inauaiiouicnt coiitrol nsed. 
Stvatofiies: 

In a mature U/i^5U system, management of t.hci nisscmination 
funL^tioub noriiiaUy can bo done through the development, use 
and control of well-developed plans. However, in an immature 
R/D5I system, this mode of management is not so feasible be- 
cause there is too much uncertainty, too many variations, etc. 
Rather, a relevant management mode is one whicli focuses on 
policy rathci tiian det(iilcd administrative management of plans 
a mode of management which steers and guides the various parts 
of the system,' a mode of management which oversees the pro- 
cess of system development and change. 

In this management mode, monitoring of the process becomes 
they key management tool and a key Agency role whether per- 
fcrmod directly or l;ontr acted out. 

D. Quitli ty Con trol 

As was noted earlier, the quality of innovative products is 
important to an effective Dissemination process. When product quality 
is low. User trust in the Dissemination source deteriorates. 

Mechan; ..iS for quality control of Uevelopmont/Production 
outputs to bo disseminated must be an integral part of tlie 
Dissemination system design. 
Stra tegies; 

The Agency role here would be to monitor the quality 
control process and to intervene to create appropriate mech- 
anisms and ])i'Oce!S:.es where these were lacking,' 
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1- . The l'rtulu(:t:/l)scivMm;ch 

ReijanUcs.'; of thu ciuality of a tjivoii proUuct, it will not bo 
of iimch v.ilno to Usu>rs if it. is not rciovant to thoir neocLs or if thoy 
arc not cupahlo of ing it. 

I)Ci;i^n Ut;f|U irouioi^: 

The Dissomlnation systoin design must provide meclianisms 
through which prodt^ts may oe adequately matched with User 
needs and capabilities. 

St_ratt)^ic^s^: * ^'^ 

Such matching can be accomplished in a variety of ways. 
For exami'le: 

1. proper identificatica of products relevant to User 
need and capability; 

2. tailoring innovations to User need and capability; 
\ 3. helping the User obtain and develop the capability 

" to implement and utilize innovations. 

It is not always necessary or wise that every component of 
a Dis.scmination system be contained within a single Dissemination 
organization. Services can be purchased or, rented, cither tempor- 
arily or regularly. For example, successful Dissemination does not 
depend upon the Dissemination agent doing its own lirinting. Whether 
it is more cost/effective to have a printing capability within an 
organisation or to purcliase printing services is a valid administrative 
question, but it is not an issue vital to successful Dissemination. 

Desi i\n Requ i remen t : , 
The Dissemination system design m^^t bo sucli as to 

allow a funding agency to treat the '''bu>v' rent" question 

as an administrative issue. 

The buy-or-rent decision may need to be orchestrated and/ 
or mandated by the Agency. 
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' • . ThciH^ is evidoiicc from a variety of sources that havinr; a '^product 
champion" is a key factor in increasing thu probnbility of successful 
Dissemination. The product champion may be a User, a Producer or 
Developer, an influential, individual or group, or even thu Dissemina- 
ting agency. Who the product champion might be can vary. What is 
' important is that the product champion be excited and committed enou|>h 
to follow through on the Di^isemination efforts. We do however note 
that prior negative User experience with product champions can be a 
. significant barrier to effective utilisation of this strategy. \ 
Design Requir ement; 

None. This is an administrative consideration. The 
design should neither mandate nor prevent the u«c of 
"Product Chaiiipions". 

For an Agency to become a "champion" may be dangerous. 
Its role is to ensure that the conditions (incentives, mechan- . 
isms) for championship exist in the field. 

Ih im ul ating User Demand 

The likelihood that an innovation will be tried, adopted, im- 
plemented, and utilized is significantly increased when there is User 
interest in and demand for the innovation. Similarly, when a relevant 
product does not already exist. User interest and demand can be very 
effective in stimulating the development of a particular innovation. 
Pes i gn Requirement : 

None. Tills is an administrative consideration. The 
design sliuuld )ieit}ier mandate nor prevent such a demand 
effect. \ 

■ 

Strat ony: 

It is often an effective strategy for a Dissemination 
agent to create conditions which wi 1 1 jrstimulate User interest 
in and demnnd for innovation. When tiiis strategy is utilized, 

\ 
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it is important that a i'muling atfcncy control the rate of 
such •'priming" so as not to poridt. .stiiiiulalion VjI" Usor 
' demands which cannot bo satisfiod, or cannot bo satisfied 
within a reasonable time frame. , 

Since the state of knowledge about Disseminv^tion (as a total 
complox process) is low, there is justification for pursuing a policy 
which permits a degree of controlled, natural i^ariat^j.on iviid provides 
mechanisms for monitoring of natural field experiments , 

■The justification for such a strategy is three- fold: 
^ 1. In conditions of immaturity and uncertainty, it is not ob- 
vious which specific mechanisms (or combinations of mechanisms) 
are most cost/effective (though certainly some building 
block elements of strategy are more effective than others 
for certain purposes) . 

2. Persons in the field may have valid insights about effective 
Dissemination methods. 

3. By using such a strate^» \ understanding of the Dissemination 
process in gcnei'al tand within a specific sector in paiti- 
cular) can be increased and, over time. Dissemination im- 
proved. 

Desiffl Requirement : 

The design must provide mechanisms through which naturtil 
field experiments can be monitored and controlled. 
St rate trios; 

1. A funding a^jency can establish a policy which provides 
support for natural experiments emerging from activities 
in tlie field. When this policy is utilized, it must be 
accompanied by monitoring and reinforcement. 

2. Monitoring is important both to control t\\Q process and 
to^learn from the procesii. Roir.t'orcemcnt is important 
because there is n premium on doing some things weli-.-excelleiit- 
ly--as compared to trying to do many things. To be avoided/ 




. V are failures Aw, to i)Our operationri which Ust^'s then Mamo 

^ ■ ■ ' on the ovcrcill I)i .ssemliiation systen, which in turn produccK 
l«nS"torm neaativo syst:omic ofl^octs. Thus when a partic- 
ular strategy (or element) is seen to work vvcll, mii. 
capability for performing that strategy has been (or i^4 
capable of being) developed, then this strategy should be . 
reinforced. 

3. What is to be avoided is, the obvious mismatch of strate- 
gies to situations. 

In spite of the premium on reinforcing successes and avoiding 
l^ailurcs, failures, will inevitably occur. There arc just too many 
joints of uncertainty and unreliability in the chain connecting Rf,D 
:o Utilization which in combination result in low success probabilities 
^urthcr, when the R/DQI system is immature, the probability of failure 
,,ecomes quite high. Thus, when the R/D^I system is immature, it be- 
comes imperative that the Dissemination system be designed to bp fail- 
safe. That is to say, if the User experiences failure in one in^^tance, 
le will be aware that other alternatives are availablo-as contrasted 
to the User seeing the Dissemination system as a monolithic system, 
.•herein the whole systo-n is deemed useless by the User when he ex» 
pericnces failure with one part of it. 

The Uisseminat ion system must be designed so as to 
provide alternative channels of dissemination to the User. 
Strategies; 

' Strategies should put a premium on redundancy, on i:iaking 

^ competitive alternatives aviilable to the User. Such 

T< 'undancy can be achieved in either of two ways: 
1. Natural Deccntrali:ied Valuation and Adaptation: When 
statural decentralized variation and adaptation are allowed, 
a vai'icty of alternatives (even redundancy) may becone 
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availablQ to tluj User, and the llsor Is thuy less likely 
to transfer his perception of weakness in ono purl, of the 
Oln.Hoinination system to the other parts. The various 
. Dinsemination mechanisms ^^hi-'jld, of course, be orchestrated 
from a higher level, but with a miniraum of visibility. 
2. A Pail-SaforCentrulizcd System Design: While a fail- 
j;. safe centralized system design is at least theoretically 

possible, dt is complex and thus very difficult to develop ^ 
and manage. If attempted, it would include: 

a. disaggregated parts; 

b. built-in competition among parts; 

c. built-in redundancy. of a few things done well. 

niSj^EMINATION IN 1HK EDUCATI O NAL CpNTISXT 
^jL.^ Lovel of Maturity of Kducational R/D^I as a jyho\c 

It has besn our contention throughout that we must understand an 
H/mi system as a whole system in order to develop relevant policies 
and strategics for f.ny one as,pect of the total system. For example, 
many Dissemination roles and mcchani:sms which arc created for an 
immature R/DQI system must change over tine or even be terminated as. 
the overall R/D5I system roaches maturity, 

In education, we find a very immature R/Df,I system. There is a 
liistory of poor quality products, which suggests the need for internal 
mechanisms for quality control. Th^jrc is a history of low levels of 
User capability in locating and utilizing products, even ih defining 
problems and identifying needs. This hi.^tory points to the need for 
intermediate mochar isms to facilitate and develop User capabilities. 

Thus, the immai.ure nature of the overall- educational R/Dfjl system 

strongly points toward the need for intermediate roles, mechanisms and 

linkages i .stitutions in the educational Uis?emination process. However 

an understanding of R/DQI systems in general also makes it clear that 

In designing a Dissemination system for an immature R/DPd system, the 
4 . / 
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dosinn ■^••wst pTov:'ulo for aUapt ability, chanyo and even t:orii>i nation oC 
certain Dissemination roles anil mochtini.sins . The imp'^rtanco of doslgninj* 
change into the system is based on the two previously montloncd con- 
flicting considerations. 

1. Some Dissemination roles and mechanisms will outlive their^ 
relevance and usefulness over time as tiio total system 
matures . 

2. • Institutions tend to strive to maintain (and even enlarge) 

their roles and their existence. 
Since such potentially desired "withering away" will not tend to 
occur naturally (or easily), the mandating of design for chanjie (o.}',., 
timed funding) and *the necessary monitoring become key Agency roles. 

2. User Setting 

If workable Dissemination strategies arc to bo developed and 
effective Dissemination systems to be designed, problems inherent in 
two characteristics of the User system must be taken into account: 
a) its enormous scale, and uj the attitudes toward chanuo likely to 
be encountered in User settings. 

A. Scale 

1. Aii_EnoiTOJusJ^bc^^ 

There are more than 19,000 LliA's in this country. Hach 
one includt^a many schools (literally, hundreds of schools in 
the larger cities). In each school there are numerous teachers- 
not to mention students (who are in a real sense the ultimate 
Users). Thus,, any idea about designing a Dissemination 
system to reach an entire universe of potential Users' is simply 
mind-boggling, especially given limited federal resources 
allocated to the educatioi; sector. 

^ • Aj;remond(^^ 

As might be expected with sucli an enormous number of 
Us.ers, there is a tremendous nmount-of diversity and variety 



among Usurs . Tlioro anr. rich and poor J)j.i-.t.r ii-.ls; urbiui, 
riubiu'ban, and rural DLstricts; jmijlic. aiul privato educational 
) systems; largo schools; atul small .schools; Distriutj; with a 

high Icvol of toacher ]>rofo}>sionalism urul Districts with a 
much lower Icvti of toacher professionalism; etc. —not to 
mention variations in educational philosophy and in the nature, 
and extent of administrative leadership. 

From tho product perspective, there are many kinds of prod- 
ucts for which there is a relatively thin (and not commercially 
profitable) market--a fact which may point to a need to develop 
special mechanisms for disseminating thin market products. 

Given tho problems of scale, it woufd seem reasonable in developing 
Dissemination strategies and mechanisms to: a) identify the categories 
ofvpcrsonnol in different kinds of User settings who have tho most impact 



on Adoption decisions, and b) focus Dissemination strategies and re- 
sources on these target Users. One possibility might be to differentiate 
U'As in terms of the professionalism of their instructional personnel. 
The educational change literature is rather inconsistent in assessing 
the i-elativc influence of teachers vs. principals and oth.or administra- 
tors on innovation Adoption decisions ( a matter rather different from 
innovation Implementation). vSome analysts emphasize the need for par- 
ticipative decisionmaking on the Selection and Adoption of new programs, 
products, etc., with a major (perhaps thy major) role for teachers. Other 
analysts urge strong administrative leadership as a preferred alternative, 
describing the real influence of teachers on such decisions as minimal 
and over-rated-"and seem to suggest that this minimal role for teachers , 
is to be viewed as desirable if innovation is to occur. What seems 
missing from these analyses is consideration of such factors as the 
professinalism of a District's teaching staff as a determinant of the 
appropriateness of one or the other of these approaches. Thus, for 
tliose Districts whore the instructional staff demonstraies a high level 
of professionalism, the appropriate Dissemination strategy would seem 




to. he to target Dif? .semination resources cHrectly at toafhor;^ . In thuno 
nlstricvs where the iii:;tructi,oiia.l staff denionstratos a consiaernbly 
lesser liOiU't'G of profossionalism, the tari\ot for DLssoiiilnation couUl bo 
Supcrlntendonts, m-inuipals, and other ailministratory — ou tlio a^-.r.uinptlon 
that the teacher}; arc likely to ''go along with" the administrator's 
do.cision. Given the limited resourceii available for Dissemination, it 
might be wise to further segment the Districts with relatively low 
teacher professionalism into those characterized by strong; vs. weak 
administrative leadership and more vs. less interest in innovation and 
work primarily with those Districts demonstrating interest in imiovation 
and stTonjj administrative leadership to promote and. support innovation. 

At times, effective Dissemination may require a stratcg>vQf^sclcc- 
tivity in the choice of User target groups -..it., which to, work. For 
example, in a diffusion model of Dissemination, the Disseminator might 
work with only a few selected User groups. To consider another example, 
given limited funding, it may be best not to try to work with a resisting 
SKA.- However, there is a danger that this kind of choice will be matl:; 
simply on the basis of convenience. Thus, it may be necessary and 
appropriate for an Agency to mandate allocations of effort and to make 
available to Disseminators information on preferred strategies. 

Resistance to innovation is probably the single most salient 

characteristic of User settings described in the change literature-- 

not only in education but in other sectors as well. In any given 

instance, such resistance may be attributable to attitudes, norms, 

and User system constraints; or to technical com|3lexities and diffi- 

' .... 

culties that make effective Implementation beyond the capabilities 
of User personnel in the absence of Implementation supports that 
are not ])rovidod; or botli (as when awareness of the technical com- 
plexities generate resistant attitudes). 

The attitudinal basis for resistance's simply that change tends 
to bo threafoning to people, in the cumiition sector, the "threat" 



catlonal Innovat i.(»n :involvt)s X?}\'>l>jAl lH>hav.ioyh; plillO" 

sopl\y chaiigo; rotihivlcat strato^' ics^ which rtqujrt) toachevii- to 
uttlj^arn "oUl" behaviors and KtratoglOH with which they ave l-a.miMav; 
etc. , 

llowevor, the -c i^h r^ug^icsti.vu cvxdeni'C tliat tho attitiulliial 
dimension my well have been given ext'.essivo stress in tho litera- 
turo and that thu technical probleni.s may in the lon^^, run have far 
greater signifiean. o for dctorminiifg the fate of innovations than 
the attitiulinal ones. More often tlvan net, teachevi^ appear to be 
willing to give an innovative idea or program a try. But very 
often, the proi;rams fall because no one has given the teacher 
needed technical support-- the kind of Implementation support 
and technical assistance that is required to make an innovation 
successful. The result is failures-failures wliich make it diffi- 
cult to introd o the next innovation into that particular scliool 
system. Tims the ensuring of proper quality control and technical 
support comi)ononts in Dissemination programs would seem to be 
of central importance. 

There has been incrcasini> recognition of the importance of 
the technical side of re!:;i stance . Thus we find Dissemination and 
Utilization being clustered or integrated in recent educational 
policy. There is validity to this clustering of the Dissominaiion 
and Utilization functions.. That is to say, it is not enough just 
tu disseminate information and make Users "aware" of an innovation. 
The User must also have, be able to obtain, or be provided, 
Implementation/Utilization support throup.hout the Adoption/Inr.talla- 
tio?T process in order to make sure thot there is the bc^t iL9i!£ii?iiL 
chance for tl\c innovation to be successful. Otherwise, tiie proba- 
bility of Uisseminatiou failure is very largG--with the attendant 
result that Disscminatioii efforts and money spent will be wasted 
and that resistance to future Dissemination will develop. 
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At least two major factorii have tentlod to create a very poor clinui^-o 
for Dissomination in tho education sector. One factor has been the lack 
of Implemcntation/Utiliaat'on support to the User during tlie Dissemination 
prbcess. (This we haye already discussed above.) Another factor is that 
many produat-s disseminated to .sc:hool Districts have been o.£ poor quality-- 
at. least some have been cloarl-y inferior to existing Uscr-dcveloped 
practices, programs, materials, etc. Tho effect of such Dissemination 
has been to create negative attitudes "tov/ard tlis iUiD syatcin and its 
outputs and also a lack of trust in both the information and outputs 
disseminated and tho sources of Dissemination information- Under such 
conditions, many Users have become skeptical of any innovation. Further, 
there has developed an attitude that anyone advocatinfr a product is doing^^ 
so for his own benofit or glorificatiojv Finally, tlie professionalism 
with which the Dissemination was pursued loft much to be desired. 

Thus, paS(t Dissemination efforts havo created a poor climate for 
Dissemination, p. climate which makes it more difficult for new Dissemina- 
tion efforts to have significant impact. 

Tho point to be made in reference to current and future Dissemina- 
tion policy and strategy is that: 

a. User trust in the source of Uissemiiiation is vital, 

b. To develop and maintain this trust, it is vital to do well 
whatever is done- -both in terms of the product and the 
Dissemination efforts. 

Pei'haps this 'is an important-part of the reason why commercial products 
are so dom:inant in school systems. . Even though commercial products tend 
not to be the outputs of rigorous, systematic RfrD, and not to have been 
carefully tested, the commercial firms do a very professional job of 
marketing jnid distributing, their products. It would seem to be the 
responsibility of NIC to ensure that similar care and professionalism is 
built into the Disseminat ioii systems it helps to create. 
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In the etlucation" sector, thoru has boon a considerable amount of 

activity that has been called Dissemination, But still, despittD all 

this apparent activity there seems to, be widespread disappointment; in 

what is porccivod to be rather minimal impact on school practices and 

programs. Relatively few RfiO outputs produced by the spcciali;u)d 

Development organizations appear to have been adopted (or if adopted, 

maintained) by the User system. Where innovative practices can be 

identified in a school systom, they tend more often tlian not to take 

tho form of locally developed ei;einplary practices or programs that are 

lost to the larger system for lack of packaging', and dissemination. 

Therefore, despite all the activity that is categorized under the 

Dis^ination rubric, there appears to have l)een relativdly^ little 

effectivG Dissemination. Dissemination programs do not appear to have 

r 

been noticeably successful in achieving their objectives. 

There would seem to be several ponaiblo explanations for this 
limited iinpact. But certainly one factor that accounts for the weakness 
of the Dissemination function in education is its fragmented, scattered 
character. Of the several typos of Dissemination activity in education 
that could be noted to underscore this point, two in particular illus- 
trate tliG point well. 

1. Deve lop ment "Add-On s": Particularly in the early years 
of federal funding of the regional laboratories and RSD 
coutcrs, Dissemination was assumed to be an activity 
that should be carried oh by each Developer, to make 
potential Users aware of and interested in implementing 
th9 Developer's own products and programs. In each case, 
a certain amount 9!- tlie total fundiiig was set aside for 
Dissemination. Such Dissemination "add-ons" tacked onto 
gi'ants or contracts for Development work appear to be 
one uL the domii^ant modes of Dissemination funding in the 
early and late 'bOs. Such Dissemination was rarely if 
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ever carried out. by spocialj.^.ed- personnel, mechanisms, or 
units with the kinds of training, backgrounds or organi^a- 
tional supports that would suggest a strong likelihood 
of success in achieving Dissemination objectives. 
2. Disscininatu^^^^ 

Stronger Dissemination efforts with some greater degree 
^ of success were (and continue to be) found within the 
domains of the various kinds of federally-funded cate- 
gorical programs (e.g., special education, handicapped 
education, vocational/occupational/career/ educations, 
ESEA Titles I and III, etc.). However, each of the 
categorical programs appears to have developed its own, 
discrete specialized Dissemination system, separate and 
apart. from any other program--its own channels, mechan- 
isms, etc. Hven if a potential User learns his way 
around the Dissemination system of a given categorical 
program to meet one specific set of needs, he is likely 
( to still be totally in the dark about the Dissemination 

I channels pote,ntially able to assist him in meeting a 

different set. of needs subsumed under a different cate- 
gorical program. 
I-Ton. tho persiKctivo of the potential User, the«, there my be 
a rmltiplicity of potential Dis.omi.atiou chmmels that mght serve hi,n. 
but he is likely to be only vasuely aware of some, totally unaware ot 
others, and too engrossed in his day-to-day operational problems to 
invest substantial time and energies in taking the initiative to cut 

through the morass. , , . , , 

Pcdoral aKencies have boco»,e inercasingly cognisant oi this problem. 
•n,e history of the OH/NIE re.,ponse over the past decade warrants some 
.n,.lv.-,is if policies are to be developed that t.ke into account the 
Iduc'atlonal context as it appears today and the vaHou., assumptions that 
„ave shaped that context in the past and my continue (in varying degrees) 
.. to do do today. 
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Recoiit^ [-Qiloml aiui State Init:laU ves ij\ DLssenu^^^^^ 

Federal Dissemination policy over the past clocaUe or i^o has 
^,one through throe identifiable emphases: first, an ompliasis on 
the. creation of , a centr-alised, comprohcnsiv'c resource base through 
which Users could" gain easy access to needed information; a 
second focus on supporting product advocacy efforts designed to 
persuade Users to adopt specific outputs of the Ri'jO system; and 
finally, a shift in emphasis to advocacy of change processes rathor 
than particular products, with accompanying provision of needed 
supports for the change process. 

^ • Creation of a^Co inprohensive Resource Base 

In the 1960s, federal policymakers came increasingly 
to recognize the scattered, fragmented character of Dis- 
semination in education and the difficulty of locating in- 
formation relevant to a vast array of potential needs, stored 
in innumerable discrete repositories (or not 'stored at all). 
The solution to these problems seemed to be to create a 
single, centralized, comprehensive, generalized (rather than 
specialized) resource base that would meet any User's needs-- 
i.c., to place all the scattered resource information in a 
central repository througli which, by the push of a button, 
the User would have immediate, awtomated access to every item 
of information available about his specified need. 

Thus, the massive ERIC system was created by OE to ac- 
quire, store, abstract, and provide easy computerized retriev- 
al of souixes from the extensive, unpublished, "fygitive" 
literature of the education sector, ERIC a) so provided 
publications that announced acquisitions to the field (and 
therefore were expected to make them more visible) , indexed 
the journal literature of the field as well as tlio^gitive^ 
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J.iteraturo storou in liUIC collection, and p::-ovided 
several hundred information ajuilysi.s products^ that, r.yr.tlie- 
si:;{;d infuriiuitiun in solectod topical aruus. 

Sov(iral chiiraatcristics o£ BRIC should be underj^cored: 

a. Iliac; was "primarily a repository for R(iyea£ch 
int'ormation. . 
. b, HRIC provided a= single, 

semination capacity. It was-' not comprised of 
separate, specialised systems targe tted at 
different User groups or focused on different 
problem areas or needs, 
c. eOUIC was a passi\'^ system. It was a passive in- 
formation repository that required User initiative 
to activato it, and therefore assumed User capacity 
to define their information needs, to learn the 
descriptors, and other tools required to make 
effective use of the HRIC system, and to screen 
and make use of the enormous quantity of (not 
always useful or high quality) output the system 
provided. 

Ke shall return to the implications of these points later in 
this analysis. * 
2. Product Advocacy 

While HRIC was being developed and expanded, a somewhat 
different Disscminatior. strategy was evolving and being pro- 
moted with federal fundii.a. The annual budgets of the federally 
funded. regional laboratories and R§D centers included specific 
allocations for Disst-mination programs, rspecially those in- 
volving dissemination of the UD outputs they were producing. 
Various institutions and mechUisms were created specifically 
to advocate the use of particular products ov prosra-.s they 
selected or developed, and to persuade Users of thoiv ncri'iSr- 
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o.g-» t;hu IriHtructlonal Materials ConttnT., ami Tit.lo III 
clo«ionr>tfation ccntorH. 

Over time, however, some tlU;f:lci»lt::i.oy in tho produf,t 
advocacy mode of Dissemination became apparent. Many of- 
tho proUucta dissominatbd wore of poor quality aiul wore 
overxcalously ••marketed" by Disseminators functioning as 
advocates. And too, much of tho Dissemination effort, 
carried out in this mode was amateurish in naturc-^it 
lacked polish and professionalism. These two factors 
together- -poor quality products and weak Dissemination 
programs- -contributed to an already negative set of User 
attitudes toward the Rf?!) system as a whole. 

3. CIianKe Process Advocacy 

NIll's policies appear to have shifted some of the 
emphasis in federal dissemination efforts, away from the 
product advocacy mode to an advocacy strategy that pro- 
vides supports for change processes and innovation in 
schools as a more general phenomenon. Instead of ad-, 
vocating particular products, strategies have been developed 
to: a) provide Users with information about the full array 
of products, programs, information sources, exemplary 
practices, etc. available to meet a given need- -comparative 
evaluating information on the alternatives where possible-- 
while b) developing tho User's capabilities for identi- 
fying needs and evaluating, selecting, adapting and imple- 
menting the products of his choice. The emphasis is two- 
fold: building User system capabilities and increasing the 
"rationality" (in the technical rather than the layman's 
sense) of Users' decisionmaking processes. Tlie State and 
federal Dissemination agencies that carry out this strategy 
provide information and supports that make it easier for 
Users to plan and implement changes (if they want to). 




Decisions on vhethor or not and how to change roinain 
with the IJsor. 

Some Specific Current Strategy neveloiyiients 

Several aspects of current Nil: Dissemination stratogy warrant 
mention. 

KRIC was designed initial 1$^ as a resource base of Research 
information. In more recent years, several initiatives have . 
expanded ERIC into an iniponnaton base for accessing a wider 
array of resources to meet User needs: R(iD products and 
descriptive product information packages; exemplary practices 
from the User system; Research and policy interpretations 
and syntliesos; guides, catalogs, prod ict inventories, etc.-- 
ali input in a forr.i compatible with the automated HRIC 
retrieval system. For those practitioners who make use of 
the system, the kinds of resourcos that have become accessible 
have broadened in scope. 

2, The. States and Interinediatc DissGmination Meclianisms" 
NIE's Dissemination policies have placed a great deal 

"I 

of emphasis on working with and through SEAs and LEAs. The 
federal role is seen as one of facilitating, coordinating, 
and providing seed money to mobilize State and local Dis- 
semination resources and to build needed Dissemination/ 
Utilization capabilities in the SfiAs and LEAs. 

Given the scale and variability within the User systei.i 
in education, and the rather limited financial resources of 
NDi, there is simply no way for the Institute to directly 
ir.ipact the User system effectively and significantly. SlEAs 
<^have substantial resource bases to apply to the Dissemination 
function, and in i-ecent years a number of SKA^ have exercised 
strung leuclorship in: developing systems for needs assessment 



and lonji-ran^c planning; taentl tying and dlssemiiuitiniT 
oxomplarv practices; and providing information .st^rvlcoH and 
tochniual assistant to local school systoms. NlH's strategy 
is doiri[»nod t\> support and strengtliou tho States in tliese 
cfforts--to provide seed monoy to facilitate and expand what 
they are currently doing, and to facilitate and expand their 
capacity to do what they are doing more effectively so as to 
bo able to coordinate education Dissendnation activities. 
• The pooling of fedcrf 1 and State Dissemination resources 
is expected to increase the potential payoff of the federal 
investment. The strategy is expected to be more cost/effective 
in the long run, and probably more effective in impact re- 
gardless of costs because of the added possibilities of 
providing services through the States that are tailored more 
adoquately to local District needs. 

NIE's State Dissemination Capacity-Building Grants have 
been used in general to establish (or strengthen) variants 
of a model that includes: a) a centralized resource, base 
providing access to the URIC system and perhaps other State 
resource bases; and b) a network of. field-based, specially 
trained Dissemination agents (or "educationar e::tension 
agents," or "information agents") who function as active, 
personalised Dissemination links between the resource base 
and the User. The various States that have created such 
systems differ in how they define the role of the extension 
agent; what services they provide; who tr.iiy are; wliero they 
are placed (e.g., in regional -type education information 
centers or in tho Districts themselves); etc. But regardless 
of which variant is used, the extension agent concept adds 
tv.'o elements to the Dissemination process: 

a. There is now an intermediate-level link to facilitate 



User access to tho contra li-ed resource base, 
b. "iivis- iK-ns .iiitoTiuodiatc link it; an act^ivo link. The 
.cxtunsjon agent works with the User to define problonii; 
and ntiods. Information and materials potentially useful 
for meeting these needs are retrieved from the resource 
base, screened, synthesized, and transformed into in- 
formation i>ackages tailored to the User's neects and 
constraints. Follow-up supports and feedback mechanisms 
also tend to be built into the system designs. In some 
cf the/liio.st active models, the extension agent may even 
"function to persuade the User of tho need for change, 
and perhaps even provide technical assistance to support^ 
the planning and implementation of the proposed changes. 
(In addition to the technical assistance capability 
that .nay be built into some of these State systems, 
some of the newer NIE initiatives such as the RJil) 
Utilization Program" provide the beginnings of a develop- 
ing technical assistance capability.) 
• We might note that all of these more recent strategies result 
from a system-level kind of thinking and that they are es- 
sentially procured to facilitate .User access to the information 
resource base. 
4. Two-WayjComim£ij£ati^ 

Recent federal Dissemination policy statements have tended 
to underscore a view of the Dissemination process as essentially 
two-way ratlicr than one-way commu|^cation' The assumption here 
is that if Dissemination efforts h^Q to. be effective in ac- 
hieving widescale Implementation and Utilii^ation of Develop- 
ment outputs, more will be needed than simply setting into 
motion one-way flows of outputs from Producers/Supplieri^ to 
Users. In addition, it is now argued, information frpil Usc^ 
KiU hav'.- to be fed back into tho systems-User perceptions of 
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their problems and needs, of the quality and effoctivenesH 
of products that are being disseminated, and of the strcnj?.thR 
and weaknesses of the Dissemination strategies and mechanisms 
peing used. Thus, one of the most recent NIM initiatives to 
strengthen the Dissemination function in education is the 
planning of' a system currently referred to as the Dissemination/ 
Fecd-Fox^varci system. 

^' A Note of Caution 

The newest Nil: Dissemination program— the planning of a 
Dissemination/Feed- Forward (D/FP) system— suggests that an 
additional "regional"- level of Dissemination mechanisms is about 
to appear on the multi-level scene of local, intermediate, State,, 
interstate, and federal agencies and organizations. Although 
still in its early planning stages, it appears that one purpose 
of the nexs- regional system will be to somehow coordinate "every- 
thing" relevant to Dissemination and Utilization that is taking 
place within a given region. The coordination intent here would 
seem to be to make the various discrete, perhaps scattered 
communication channels and resources in an area more accessible 
to the User. 

There is a potential danger here, one to which we have al- 
j ready referred. Certain typos of coordination and integration 

\ may indeed need to occur. " But if there is too much "coordination," 

redundancy may be eliminated-"and with redunancy, some of the 
competitive alternatives available to the User will be eliminated. 

Given the likelihood of ^failure in the education Dissemination 
system (as we noted earlier), the Dissemination system needs to 
bo made fail -safe. NIE should not create a s iiiRl e ,^ e xc 1 us i ve , 
intermediate Dissemination channel to link into aN^'iAien User 
setting. If alj_ User information in a region must channel through 
a' regional a^jency and the regional agency fails or is ineffective 
with the User, then there is the potential that; the whole Dis- 
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i^osnination sysf.Rm way be viewed ay uaeloss iHH-.ause the User has 
no pvilatablo alternatives. ; . 

If such royionrtl aaoncies have to hv Uevolopovl (for political 
or other ratiotifil roasoi^s oi^ regional needs and opportunitloiO , then 
this system sho.uUl bo doaigncd in a way that does not. eiinvLnuto 
diversity, docs'iiot result ?ri too much standardif.ation, does mt_ 
eliminate alternative infornuu.ion access channels. A poBsible 
solution might be to take tlie "i^gionul opportuivity" to create 
iri,L;>yd, an alternative system to that pursued by the States, but 
wcrking in coordination with them. 

The orchestration and coordination which is necessary in upper 
levels of the Dissemination system (i.e., in the Agency) nocdr^ 
to be subtle. To acliieve such subtle yet effective orchestration 
and coordination is difficult because the natural tendency is to 
overccordinate and overmanage— a tendency that can kill off the 
\Qvy dlvarsity that is the essence of a fail-safe system design. 
Nlf, will need to be actively involved (and very well informed as 
to the critical issues and impacts) in the system desigii and 
functioning. 

Personnel Base 



We noted earlier how critical it is for Dissemination strategies 
to be carried out in a highly professional manner. In education, however, 
a well developed personnel base of trained Dissemination specialists 
is lacking. Most of those carrying out Dissemination activities ^appear 
to be practitioners by training. They are proceeding intuitively and 
learning their jobs through hard, often unsuccessful, experience. Several 
federally funded programs have been developed in recent years for training 
D{iU (Dissemination and Utilization) specialists. Dissemination mechanisms, 
however, are expanding far more rapidly and creating a far greater demand 
for trained personnel than these programs could ever hope to keep Up with. 
This problem suggests policy options in need of considoration«--c.g. . 
e.xpanJir,g the training capability and the size of training progtnms to 
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moct tiic dciand for tralnetl mii j>tn\sonnel, or slowing ithe rale of Die- • 
semr.niit'ion .^y^nvU) oxpaiK-iion. 

If NHi. doci'Je.s on a policy of focilitatinp. a?, raplvl an oxpan;;iau 
ot* the ^^«•I^S(>nael base as pOh.sihle— to meet the anticipated expansion 
rate of tho D:lsi?oiulnation/Utiliaaiion function— the Instituto may luivo 
to ci\n.sidor dove.lop'in^ ruoruitniont incontivos an well at? supporting 
expanifi,on of thu m\) training capability. In this regard, two points 
can be suggested: 

a. Personnel with classroom and scliool system backgrounds may 
ho particularly woU suited for DI:,U positions. 'Hiey arc 

. likely to be sensitive to User needs and constraints, r.ve, 
likely to interact well with Users, and, in comparison ■'i.i- 
many others who might bo recruited for thosvs positions, 
are more likuly to view this work as important and exciting. 
An additional consideration hero would seem to be tho large 
ro5-:?rvoir of such talent available in the sizeable teacher 
population currently unemployed. 

b. Tho other likely source for recruitment would soom to be 
persons trained and employed in universities. However, 
there would seem to be a strong likelihood that 3uch per- 
sons would be oriented more toward an RfiD perspective than 
the User viewpoint, would be less sensitive to User needs 
and constraints, less effective in interacting with Users, 
and less likely to find D^U work appealing or exciting. 
Clearly, there are bound to be exceptions, and talented Dis- 
semination agents may be recruited from a wide range of 
aotcings. But if recruitment efforts are to be adequately y 
focused and concentrated, choices will have to be made about 

t. 

where the stroiigust payofi' is likely to be. To meet this 
noud, we would .su^jgest the User setting as the most appropri- 
ate focus for recruitment efforts. 

7 • The Inst i tu tj_o nj.1 Base 

Within the educalion sector, one finds an enornuius number and 



tlu\.:;.<.ity of orgnniiiat ioiir^ iuvolvcHi in i-arryim; out work rohwanl lo 
l).b-;r .^T.i nation una Uti I i mutton. ThoHo orgua.r.'.nii.oiu> will have to ho 
takon iu'uo uvscouut., faeiUtalca, I'.nd iiudiitored li tlu; U.l 9 semi nation 
function- in the cdr.cution yyirtuni is to bo ut't'ectivc. 

i The enormity aiul comploxlty of thl.s iiistitutional bdso for U.ls- 
ji'einination can be soon by simply dovolopinf, a iiat: of Bi.ich org^i»i"-^tions . 
Vov cxami)lo: 

1. All Producers 

2. Commercial firms 

Regional laboratories and U^J) centers 

4. I'ederally funded programy having a Distjeiuination component 

5. SliAs and ISAs (Intormediatc Service As^oncies) 

6. Houndatluns and universities to some degree (porhaps not 
very effectively, and usually in priiit form) 
Cor.scj^tia of various types 

S. Networks of Users (usually created by the producer of a 

specific product) 
9. Recent NIG funded programs which have created various 

technical assistance mechanisms 
10. Various sorts of organii:ations that are focusing on \W) 

Utilization (e.g.: Developers/Users; SEAs/Uscrs; User/User;- 

etc.) 

These multitudinous and varied organizations mugt be taken into account 
simply because they are part of the Dissemination system, even if they 
arc not all part of the system created by the federal government . As 
in connection with various- other aspects of the W/mi systam, NIH may 
have to accept the initiating role in orchostruting this diverse but 
relovant commuHity, especially if NIH follows (as it should) a strategy 
of buildinfi, as feasible, on what .already exists in the field. 

"v 
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(MvHiu tiu3 i.mm:it;urit:y oi' f.ljo eduaat.louul U/nGi system, the 
large. scaU aiurvai.lal)U;lty of tho Uyor 8yF,t;om, and t;ha frag- 
'j^unted nature oi; current. Hi abcminat ion. efforts, tho key I'olo 
f'^vr NiH would 5'.(ji>m to be one of system orchestration. These 
vcixy uharactovi.?5tiCi» call fof some dO'',roc of hiahci* (i.tJo, 
federal Agonuy) level orchestration. Beyond this, Dissemina" 
.*ti.on is a systems function and can he .iiiaiiaged only at an over- 
all systems level. Althoui;h Oli clearly has some rcsporisibili- 
tiey in carrying out the Disycmination function in education, 
NIH is tho lead Ai^ency for Uoseai?ch and Dovclopincnt in education 
and is more likely to view U/t)i;I f^n^tio"^^ i« intorrolated 
sy.?t:eiu3 terms. Close coordination with Oli will undoubt.edly be 
needed.. But if NIH is to carry out its legislative mandates 
to both "build an effective lUH) system" and "improve practice," 
Nil; leadership in carrying out tho orchestration rolo will 
clearly be necessary. 

This kind of orchestration required by the education sector's 
immature, lar«e and fragmonted Dissemination system must be 
carried out in a proactive Caa compared to a reactive or passive) 
mode. Orchestration ruquirus an active posturo-~it will not 
happen if the Agency takes a passive stance; and it is. likely to 
take dysfunctional form if the A^jency behaves in a reactive 
mannv^r. 

At the outset, Agency planning must be based on a clear 
unUerstandinp, oi" what already uxists in tho ficld--what is being 
disseminated; by wiiom; to whom; how well; and with what dogrce 
of success as measured by User system Adoption, Implementation, 
and long term Utilination/Maintcnance. Some of this information 
already exists--lii scattered sources, reports, and in people's 
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hendH (and lilus) across the multi-level federal, State, inter- 
rwdiate ami local sy.stcra. vSo:na of it is bcin*; gathered in the 
curront n/y-T system planning effort. However this information is' 
gathered- -from co:itracted studios and analyses, througli |ield in- 
put, .as. an outcome . of the creation and orchestration of a State/ 
regional/national organizational mechanism for Diysemination co-^ 
ordination- -it must then bo used by the Agency,, taking, a somewhat 
directive approach initially, to make decisions (with Cield 
guidance and advice) as to directions, objectives, etc;- for Agency 
orchestration of system facilitation and system building. Given 
the long-term system-building, concerns of the Institute, orcho3- 
Itration will have to take on one , additional dimGnaion--i .e. , 
balancing coordination of current system needs witn long-term 
system-building requirements . 

5. System Facilitation and System building 

Throughout our analysis,, we have emphasised system facilita- 
tion and system building as 'the two' overriding objectives of NIB 
Dissemination policy. In the short term, the basic nfeed would 
seem to be: a) to facilitate existing insseminatioia--ef forts and 
activities^ and b) to fill "gaps" in the system/ either by 
creating new institutions, roles, and mechanisms or by. renting 
services. Over the long run, the need is clearly one of system 
building.. \ 

"^^ NIE-initiared system'design must be congruent with the state 
of the'^overall Disseminatioh 'system (and the educational R/D^I o^' 
system more broadly). Currently, this me£>ns facilitation and 
gap-filling. Over time, this means building into the system the 
capability to adapt, change ^und even terminate component parts 
as the overall system matures. 

If the system is to function effectively, it must bo mcuiaged 
in a way that avoids dysfunctional disruptions and permits Dis- 
si^nination agents to gain the confidence and competency which 
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voiae f'roni c\sperionce ami faiiuliar.lty. CoiiBO(.juoiU'.ly, what is 
noovlod i.s both jitahility jukI ciianao. To remain congruent with 
overall systtmi iiiaturUy over tbm, Nin-,ln:Uiat;od DUsornlmition 
i.icohsnisiiis must change. Still, a system amy t have enough 
stability to permit it to curry out its functions of foctively-- 
to buiUl tyur*t,'to avoid confusion, et^r. Tims, stability too 
mubt be built into the system dosign, and must bo mo\iitorGd 
and attended to with great care. \ 

C. Process Mo do o f Ma nagcmont Goutrol \ 

With regard to the Dissemination function (in contract, tor 
instantjo, to Research) we have suggested that NIU must ho 
directive in cnrryirg out substantial portions of its overall 
role. Wliothor to control is not the issuo. Rather tho issue 
is Kliat to control. And here, we have focused on th.-^ process 
.•nodv of managrnent as most appi'opriate . 

Inhere the overall system is weak and varied, only the pro- 
cess mode of management would s^em to be feasible. Where the 
overall system is strong, only the process mode is needed. Cur- 
rently, then, the mixture needed would seem to be one of: 

a. process control over weak areas of system functioning; 

b. process control through collaborative coordina.-.ion 
where strong Dissemination agencies exist, 

Li n kat^es/Trans forms 

IVe have noted repeatedly throuj^hout our analysis how weak the 
linkages (or transforms) are betv^'een stages of tho R/Di^I process 
in education. Given the weakness of the knowledge and t^schnology 
base of the field, it is not entirely clear how to transform 
Research findings ' o Development products (or even how to trans- 
form tile accumulating Development state of the art into Develop- 
ment products); how quality contrul is to be applied to Development 
outputs; iiow Development outputs are to be di sbomlnatod to Users; 
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hoK aissominated proaucf.s are to be used; or cvyn how to 
transluto Us-sr prohlems and noeda into rosearcliablo questioiii; 
or Dcvolop:nant product spccification.s or I)i.ssoniiiiat:lon/Uti llza- 
tion strar.ugy spocifications. 

■ Theso weak linkasos or transforms must becomo a major focus 
of Aftoiicy attention if the field's kuowledse/tedmology base and 
its R/Wil system arc to mature. With. regard to the Dissomination 
function, this suggosts the need for focused effort on: 

a. designing quality control mochanisms for screening 
Research and Development outputs prior to their 
.dissicminatlon; 

b. improving the design of Diysemination mechanisms ami 
strategies through funding analyses of tlie requirements 
for optimizing User/product/Dissomination strategy 
"fits"; 

c. strengthening the Dissemination/Utilization linkage 
through funding analyses of m,U roquiromonts for given 
product and User types and perhaps too dociiiuentation- 
and-annlysis Research on naturally occurring variation 
or planned variation experiments in different patterns 
of DSU activity; 

d. design of Researcher/ Us or and Developer/User linkage 
mechanisms that might strengthen the transforms between 
User needs, problems, preferences, and constraints, on 
the one hand, and, on the other hand, initial Nopd Iden- 
tification for a given Problem- Focused Research project, 
onooinn Need Identification (and User/Product "fit") over 
the course' of Hcvelopmcnt projects, and the design and 
i.iiplcmontat ion of Disscrninat ion/Ut i Uzation strategics. 

H. Use of_ M i X e d S t r a t e g i e?. * 

Since the state of Icijowlodge about Dissemination ("as a total 
.-omplex process) is low, it seems most appropri;ite for the Agency 
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to pursue a policy which: 

a. permits a substantial degree of controlled, natural 
variation; aind / 

b. provides mechanisms for monitoring pktural fiel ' experi- 
ments and building-in documentatiph-and-analysis types 
of Research projects to enable us to develop a cumula- 

^ tive knowledge/ techno logy base from these experiments. 

fn addition, g^^en the uncertainty and unreliability df our 
undei^standing of DSU processes at this time, and the strong proba 
bility of a significant number of failures, it seems essential to 
provide Users/with alternative channels of access to the avail- 
able resourofe base. Consequently, we have emphasized thp need • 

for: ■ • " ■ 

a. decentralized rather than monolithic, centralized, 
"over-coordinated" approaches; and 

b. fail-safe system designs that permit a substantial amount 
of redundancy. 

Furthermore, given the significant amount of variation in 
both products and Dissemination modes in education, there would 
seem to be additional support for an overall Agency orientation 
toward supporting a range of alternatives as well as combinations 
(i.e., mixed strategies). 

F. Control Growth and Expansion R ates 

Since the Dissemination system can be built at a faster rate 
than User ability to Adopt, Implement, and Utilize what is dis- 
seminated, we have underscored the danger in creating and/or 
expanding Dissemination systems too quickly. Since such large 
and expensive systems are likely to generate unrealistic ex- 
pectations by Users, Congress, and NIB of immediate and widescale 
benefits, the size 'and expansion rates of NTE-sponsored Dissemina 
tion systems must be controlled carefully. Such systems should 
he no larger, and should be expanded no more rajildly, thnn is 



congruent with insuring: 

; a. adequate quality control of outputs disseminated; 

b. needed technical support to increase the likelihood of 

♦ 

User success with disseminated outputs; 

c. a high level of professionalism in all Dissemination 

efforts; and V 

d. adequate recruitment and training capabilities to permit 

the trained personnel base to expand at a rate congruent 
with the rate of system expansion. 

C« Monitoring 

-If NIB is to effectively carry out it;s orchestration role, and 

is to manage the system wherever possible through fef iback and 
field guidance,, the Agency must put into place a carefully design- 
ed monitoring system, with monitoring indicators keyed to the 
kinds of pitfalls in system design and implementation we have 
suggested. The monitoring system must be designed well. And 
, it must be used in nn ongoing, rou^inized manner to insure con- 
tinual monitoring, feedback, and a4aptation-.the virtually im- 
perceptible kinds of minor "fine-ti^nVng" adaptations, rather than 
the reactive dysfunctional ly major Byh^m modifications more 
likely to be the -outcome of episodic, di^continous, only occasion- 
al monitoring. 
H. Mandatable Features 

There are several aspects of Dissemination functioning v^hich 
NIE should feel reasonably free to mandate in a somewhat direc- 
tive manner. For example: 

a-. NIE might mandate specific allocations of efforts to 
particular thin market products or to dissemination 
to meet the needs of specific groups not well served by 
■'' existing mechanisms. 

b. It v.ould probably be unwise for NIE to mandate the dis- 
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semination of specific products. But it would seem 
perfectly reasonable for NIE to facilitate, or even man- 
date, the creation of conditions (incentives, mechanisms) 
that would make it possible for, or even encourage. 
Producers or Users to become Product Champions for products 
of their choice. 

c. While the decision to buy or rent services may be an ad- 
ministrative matter, conditions that would make "renting" 
a feasible option in certain cases may need to be facili- 
tated. This facilitation may require some Agency orches- 
tration or even mandating. 

d. NIB might mandate certain decisions as to who is to func- . 
tion as the Disseminator for a given NIE-sponsored project-- 
e.g., whether a given product is to be disseminated by 

its Developer/Producer or by some specialized Dissemination 
mechanism. 

2. NIE Skills Required j 

We have made a number of points thi^oughout this analysis that 
suggest the need for NIE to possess and to exercise great skill in its 
management of the Dissemination function. We have underscored the con- 
siderable complexity of Dissemination as a total process, regardless of 
the particular sector in which it is carried out. And we have noted 
how the multiplicity of institutional bases for Dissemination in the 
education context, as well as the multiple levels of government involved, 
complicates that picture manifold for NIE. What the Institute needs/ 
then, is extensive familiarity with what exists now, and how well it 
works, as well as some rather clear notions about the directions it 
might take in system facilitation and system building. 

NIE staff will need to draw on a high level of professionalism on 
matters affecting Dissemination system functioning. This professional- 
ism must come from its own internal staff capabilities and the capa- 
bilities it rents from various consultants drawn from academic, R^U, 
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and User (SEA/ ISA/LEA) settings. 

Of'^all the Kinds of skills Agency personnel must have to carry out 

IIE's role in the Dissemination system most complex of all are the 
inds of subtle raanagoment skills required of personnel who must func- 
tion as "hands-off" planners--subtly, almost imperceiitibly coordinating 
and orchestrating at' a higher level a system that is highly decentral- 
ized and redundant and that is performing a critical system linkage 
role in an immature R/DQI system in which most of the parts to be linked 
are weak and underdeveloped. We do not mean to suggest that this will 
,,be anything other than an incredibly difficult, almost impossible task 
•requiring skills that probably do not as yet exist— within the Agency 
.:X>r., outside. 

C - Still, if our analysis is accepted, this is the Agency role re- 
quired and these would. seem to be the kinds of Agency management skills 
to be developed. How such skills might be developed is another matter, 
outside the scope of this analysis. Clearly, though, if NIE is to 
function in the manner we have suggested, the matter would seem to be 
one that merits some attention. 
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EVALUATION RESEARCH ^ 
I« THE NATURE OF EVALUATION R^EARflH ^ 
1. Defining Evaluation Research 

A. Evalu ation Resaarch; Evaluation and Research 

We must note at the beginning of this functional analysis that 
we have chosen to use Xhe term "Ev^luatW Research" inateaa of tho 
more connon term "Evaluation". We think this is an importawt dis-. 
tlnction. The term "fivaluatlon" leads one\o focus on the process 
of analysis and interpretation. While this is indeed probably the 
typical understanding of this function, such aii understand ius is 
seriously flawed. "Analysis" and "interpretation" require a data 
base that can be obtained only by some form of He^earch process. 
Thus, this function must be understood as being an interactive 
combination of Evaluation and Research — thus our choice ol the term 
"Evaluation Research". 

. B. A Unique Funct ion; Knowledge Production and Knowledg e 
* Utilization Characteristics 

The Evaluation Research function is a somewhat unique function. 
On the one hand, as Research it will tend to have characteristics 
normally associated with the Knowledge Production end of the Inno- 
vation process. On the other hand, because Evaluation Research 
is usually carried out for the benefit of Users (administratf>rs 
and policy makers) and because Evaluation Research cannot (by de- 
finition) be performed until after knowledge has been utilisied, 
Evaluation Research .will tend to have characteristics normally asso- 
ciated with the Knowledge Utilization end of the innovation process. " 

Figure 4 illustrates the KP/KU nature of Evaluation Research. 
Figure 4 also indicates that there are different types and purposes 
of Evaluation Research (as we shall note later). Because this 
unique combination of KP/KU characteristics underlies many of 
the tensions associated with the Evaluation Research function, 
it is vital that this be understood. 
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2. Two Types of Evaluation Research; Formative and Summative 



For many analytical purposes It has been thought useful to see 



Evaluation Research as having two basic forms (though we realize there 
la > some overlap) • 



A> Formative Evaluation Research 

Evaluation Research findings may be generated at Intervals during 
the life of the program In order to provide a monitor Ing/self -correcting 
function. As the program will not have been completed » formative 
Evaluation Research findings must be considered to be tentative ~ 
but they are nonetheless Important to program cidmlnlstrators. Their 
function must be seen primarily as part of the program administrator's 
or developer's team. Seen this way» as helpers and not information sources 
for external evaluators» formative Evaluation Researchers need not (should 
not), pose a threat to program personnel. 

B. Smnmatlve Evaluation Research 

The basic purpose of summative Evaluation Research is to determine the 
effects of a program through some form of pre/post or time-series or similar 
types of analysis. In this case, findings are generated after the completion 
of a program (or at least, for ongoing or long-term programs, after the pro- 
gram has been in existence for a sufficiently lengthy period of time to 
permit pre/post, analysis) . 

This pre/post summative Evaluation Research may be performed in order 
tc^determine hoi^well a program has met its objectives and/or to provide 
a cost /benefit analysis. The change-detecting summative evaluation nay 
also be done simply to Identify and understand the effects of a program, 
apart from program objectives and cost/benefit considerltions* For 
whatever reason it is carried out, summative Evaluation Research involves 
measurement aimed at detecting changes or impacts attributable to the 
program. \ A 
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Figure 4: The Multiple KP/KU Characteristics of Evaluatio n Research 
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C. . A Research Dlleiaaa 

It irS Inportant now to note that there Is an Inherent tension in thd 
purposes it formative and suiq^ltttive Evaluation Research. 

Format ive*E valuation Research seeks to provide evaluative information 
in order that mifications land changes (if needed) may be made in the 
program— actions which break the rules of Research "control". 

Susnative Evaluation Research, on the other hand, seeks to determine 
the effects of a program without modifications and changes which would 
Inkroduce "impurities" and reduce Research "control" * 

Persons familiar with Research will recognize that the above com"* 
parison of formative and summative Evaluation Research is somewhat over- 
stated but nonetheless reflects a very real Research dilemma. It is our 
purpose her? siniply to take note of this potential tension involved in 
Evaluation Research, not to provide a detailed discussion of this issue, 
an issue that we know has been well recognized in the field in general, 
and by NIE and OE in particular. 

D. The Same or Different Evaluation Researchers ? 

One related issue must also be noted here. In those programs for 
which there is to be both iEormative and supnative Evaluation Research, 
many have questioned whether the same Researcher can properly do both. 
T^e issue here is two-fold. First, the performance of formative Evaluation 
Research will require some degaee of Researcher contact program per- 
sonnel at various times during the life of the program. This contact 
can lead the Researcher to develop interpersonal relationships with pro- 
gram personnel and Researcher "interest" in the program— factors which 
raise the question ab ut the Researcher's capability tp provide objective 
smmnative Evaluation Research, This problem inMolves not only Research 
considerations but cou^be and is often raised for political purposes. 

Second, depending upon «k«ature of the formative Evaluation Re- 
search design, it may be reasonable to' expect the formative T^esearcher to 
have interpersonal and coniaunication (and perhaps design) skills which are 
not needed by the sunmative Researcher. 

With' thes^ conditions in mind, consideration might be (and sometimes 
is) given to the use of separate formative and summative Researchers. On 
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the other hand, such a strategy would require extra coordination and 
likely be more costly. More typically, in federal social service 
programs however, only sunanative evaluatipn research has been required, 
and the formative aspects have been ignored. 

3. Three Purposes of Svaluation Research 

To further understand the nature of Evaluation Research and %he 
tensions associated with it, we must also be aware that Evaluation 
Research has three basic purposes (or uses) — > purposes which are 
to some d agree inherently in conflict. . 

A. For Policy Makers and Funders 

One purpose of Evaluation Research is to provide the policy 
makers with information about a program (or project) as a basis for 
decisions concerning future programs* Are the program's results 
(compared to program costs and objectives) of sufficient merit to 
warrant program continuation and/ or expansion? Or- should it be dropped? 
Can/should it be modified? If so, in what ways? Can the program be 
used elsewhere? As is, or in mo<^ified form? Does the evaluation of 
this program, provide insights about other (similar) programs? As can 
be seen, these are post-program questions that are best answered by 
summative Evaluation Research. We note also, however, that in order 
to monitor .a program, policy makers and funders may also want infor- 
mation provided by formative Evaluation Research. 

B> lor Proeram Administrators 

A second purpose o£ Evaluation Research would be to provide 
program administrators with information upon which to base possible in- 
process program modifications and changes. Thus, program administrators 
would tend to be very interested in formative Evaluation Research — • and 
not so concerned with its effects on summative Evaluation Research. 

Indeed, Evaluation Research (hott( formative and summative) are 
systeim functions — they are a basic part o;^good system operating 
manag/ment. The program being designed and implemented should have 
the benefit of what can be learned as the process proceeds. It is also 
vital to have a measure of the impact with the needs of redesign, im- 
provement and future efforts in mind. It is unfortunate that summative 
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evaluation Is too often associated vlth a threat to program personnel, 
80 preventing its' proper exploitation as an aid to the process. 

C. Knowledge Production 

Simply because It is Research, Evaluation Research can add 
tQ the knowledge base of the field. However, as this purpose has long^^ . 
• term rather than immediate impact, this potential system-building value 
can easily be lost simply by being overlooked by policy makers and fund- 
ers (whose concerns in Evaluation ^Research tend to be jjj^re immediate). 
This value can be lost through inadequate development of the Research 
design, through the selection of less competent Research personnel or 
of "skilled" Evaluation Research personnel who have little interest in 
the substance of the phenomenon being evaluated, and through failure to 
disseminate the findings. -^It must now be emphasized that the overarching 
purpose of Evaluation Research is to reduce the uncertainty of decision 
makers — perhaps most specifically the policy makers/ f under s who have 
ultimate control .and decision power, byt also the program administrators. 

4. The Conflict Potential Among Significant Participants 

As the above discussion has Implied, there are at least t^^ree 
significant participants in the Evaluation Research process: the sponsoring 
policy maker s/f under s; the program administrators; the Evaluation Re- 
searchers. The program staff could be added to this list. Because of 
the value-lader political context of education which we discuss later, 
we must also add the public. Each of these participants will have dif- 
fering, interests and viewpoints about the Evaluation Research which 
may be in conflict. ' 

We have already noted the potential for conflict between the 
interest of admrnistrators In formative Evaluation Research and the 
interest of Evaluation Researchers and policy makers/ funders in summatlve 
Evaluation Research. The Evaluation Research literature notss a further 
potential conflict between administrators and Evaluation Researchers. 
Comments that admiriistrators tend to be concerned with organizational 
stability and survival Issues while Evaluation Researchers introduce 
organizational change and growth Issues are common, t^hlle both concerns 
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are valid orgianizational goals, the poteatlal for "threat" to the 
administrator exists thereby buildiRg in a potential subversion 



related either to a product which is disseminated into the real 
world or to a program designed to provide people or organizations 
with services » 

Evaluation Research is directly related to decision making pro- 
cesses — either as corrective feedback for administrators or as 
information which can influence policy/ funding decisions • This 
means that Evaluation Research has an "immediate" time frame in terms 
of usage^ ^ 

The management of what is being evaluated (i.e., a product or 
program) is always in the hands of (or at least influenced by) 
someone other than the Researcher. 

As we will note in more detail later, these Knowledge Utilization 
characteristics of Evaluation Research are in (often sharp) contrast 
with the Knowledge Production characteristics o£ Basic and Problem-- 
Focused Research^ Knowledge Production types of Research are de- 
signed for the specific purpose OC^ producing knowledge and may be 
considered important in there own right, apart from any' potential real- 
world impact. By contrast, the concern of Evaluation Research is 
specifically with the real-world' impact both of its findings and 
of the product or program oeing e\^luated; and knowledge pro- 
duction per se must -be considered a secondary purpose (though im- 
portant, as we have noted). 

We will later note that this "real^-world context" has some im- 
plications for the skills and experience required of personnel. 

6. The Program Focus of Evaluation Research 

Evaluation Research may be performed in relation either to 
products? or to prq^rams. However, mc/st often unlter government fund- 
ing. Evaluation Research is focused on more amorphous human service 
••programs". Consequently, we shall concentrate the ri^st of our 




Evaluation Research takes place ip a real-life context* It is 
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analysis on that Evaluation Research vhich is targeted at such social 
or human service programs* 

A* A Specific Focus 

Evaluation Research must focus on a prograis as the program actually 
is, not as might be suggested by the Researcher or the Research literature* 
Thus, there are sigiiificant constraints on the Evaluation Research process 
In terms of Evaluation Research goals, design, time frame, data avail- 
ability, etc. constraints which Researchers tend both not to under- 
stand and not to accept. This fact can be an impediment for .attracting 
competent Researchers. 

B# Program Goals 

Program goals r.ovide the basis upon which a Research design is 
developed and evaluaLion is made. However, program goals may not be 
simple and clear cut; they may even be the wrong goals. When such 
conditions exist, the Evaluation Researcher would be confronted with 
two significant- issues. First, designing the Evaluation Research will 
be difficult at best* Second, there would be the issue of the proper 
role of the Evaluation Researcher with regard to, the possibility of 
suggesting changes in program goals. We shall comment further on this 
issue later. 

C. Program Complexity 

Programs are rarely simple, especially in human service fields 
such as education. For example, both the content and the outcomes of 
a program may depend as much on the competency and interests of program 

' ntrators and program staff (or on how well the program has been 
coa nicated to them) as in the nature of the program^ itself. This 
fact, increases significantly the complexity and difficulty of Evaluation 
Research. 

Similarly, programs will vary. A program may be "broad-aimed" 
or narrow in focus. Where a program is being carried out by different 
groups in different places and contexts, local vari-ation may be per- 
mitted (or even encouraged) as for example in the Follow-Through program. 
Indeed local variation may exist even without "official ^ anction." 
Where local variation does exist, it will be difficult to provide an 
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Evaluation Research design which can validly aggregate dissimilar 
data* Thus, there is^NaLflariger that ^Evaluation Research will be 
based upon the "lowest common (and probably meaningless) denominator." 

D. The Changing Nature of Programs 

Human service programs are "change -oriented". That is, the \ 
effectiveness o£ human service pi grams is judged by the magnitude 
or significance of changes in human behavior or attitudes, in social 
settings, In community conditions, etc. The programs themselves " . 
tend\to change because of feedback from participants, from Evaluation 
Research, from. environmental forces. Thus the Evaluation Research 
design and methodology requirements may change. Questions of data 
comparability and aggregation may arise. Additionally, there will 
be the control issue of whether the chan^ being measured results ' 
from the program or from other organizational or environmental fac- 
tors. 

E. Research SUillst A Methodological Issue 

For Evaluation Research, different Researcher skills may 
be needed for different kinds of programs. Specifically, x<here a 
program area is reasonably well understood and the data needs, para- 
meters, appropriate Evaluation Research designs etc. are reasonably 
clear, it would generally be possible for the Evaluation Research to 
be done by Researchers who are competent in standard Research method- 
ology but who have minimal understanding of the program area itself. 

However, where the program area is of an innovative nature, 
where the program area is not well understood, or where it is not 
clear what are the data needs, the parameters and the appropriate 
Evaluation Research designs, it would be imperative for the Evalu- 
ation Researcher to be quite familiar with the program area and to 
have skills which would enable -^lim to create the appropriate Evaluation 
Research design. 

7. Key Characteristics of the Evaluation Research Function 

At the beginning of this functional analysis, we emphasized 
that Research is a basic component of Evaluation Research. Thus, we 



would expect to find similf^rities between Evaluation Research and Basic/ 
Problen^Focused Research. However, we also noted that Evaluation Research 
has characteristics normally associated with the Knowledge Utilization 
and innovation process. Thus we would expect to find dissimilarities 
betwean Evaluation Research and Basic/ProbXem^Focused Research. The program 
focus of Evaluation Research should also lead to differences in comparison 
with Basic /Problem -Reused Research. 

3hus, it is Important to understand the characteristics of the 
Evaluatloa Rataarch function and to understand the similarities and di£- 
ferencea between Evaluation Research and Basic /Problem-Focused Research. 

A. The Technology Base; Design and Methodologies 

Evaluation Research and Baslc/Problem-Focused Research are in 
many respects quite similar in terms of their design and methodology (the 
technological base). While there are differences, there would seem to 
be more similarities than differences. This is especially true if we 
Include within the category of "standard" Research design and methodology 
.the process-oriented methods of fields like anthropology and the economist's 
analytical approachvwhlch Is increasingly used in cost/benefit studies. 
Thus much of the distinction between designs and methods used in Evaluation 
Research and in Baslc/Problem-Focused Research appear to involve points 
of emphasis and matters of design implementation. 

B. Purposes j 

We have already noted that Evaluation Research lyas three possible 
purposes: providing information to program policymakers/funders; providiag 
information. Jto program administr'ators; and knowledge production (i.e. ,. adding 
•to the knowledge ba&e of the field). However, we have also emphasized" that 
the overarching purpose must be- the providing of Information that reduces 
decision uncertainty for decision makers. 'Thus, in. direct contrast to- Basic/ 
Problem-Focused Research, knowledge production per sq is a secondary purpose. 

C. Problem Definition 

A basic issue in any Research is control over problem definition — 
i.e., determining what is to be researched. In Basic Research, it is 
clear that the Researcher must define the Research problem. In Problem- 
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^Focused Reseai^ch, there needs to be User /f under input, but the Re- 
searcher is still predominant in translating User/£under problem 
statements into researchable questions. 

In Evaluation Research » the decision maker rather than the 
Evaluation Researcher is predominant in problem definition. /Indeed 
we find that funders often define the Evaluation Research p^blem 

^before procuring the services of an Evaluation Researcher. 

In this context, the Issue may arise as to what is a proper 
role for the 'Evaluation Researcher when ha rceives th&t the problem 
definition of the decision maker is Inadequate — i.e., too vague; 
too narrow; would prejudice the outcome; inappropriate for the 
specific program j etc. ^ Must the Evaluation Researcher assume that 
he is an "employee" who merely accepts his "employer's" problem de- 
finition and statement of program objectives, carries out the task 
specified by his "employer", and then simply provides the "effective- 
ness, assessment" ujpon demand? 

Or is the Evaluation Researcher to assume the kind of "con- 
sultant/client" relationship with the decision makers that would 
allow the Researcher to guide his "client" in developing the problem 
definition and to assess the appropriateness of program rationales, 
objectives and strategies (in addition to measuring program effective- 
ness)? The answers to these questions will significantly affect the 
Evaluation Research process from the initial design stage to the 
reported findings. It appears that strong leaders in the Evaluation 
Research field opt for the "consultant /client" relationship, arguing 
that the acceptance of an "employee" type of role by Evaluation Re- 
searchers is a major reason why so much Evaluation Research has been 
of poor quality and-^ias been unable to answer the questions asked. 

D. Clear Targets 

In contrast to Basic and Problem-Focused Research, the "carget" 
of Evaluation Research is clear. That is, whatever the decision 
maker says is to be evaluated is the focus or target for Evaluation 
Research. The Researcher may or may not have been involved in setting 
the ^rget, but there will nonetheless be a specified target. 
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E. Tima Frame Cons*.ralnts 

lo Evaluation Research, the time frame is specific (being de- 
fined by Che deciiion maker) and relatively narrow (in* order to 
provide tijnely information to decision .makers) . This is in contrast 
to Basic/Problem-Focused Research and is likely to be a source of 
tension and fru3jtration to Researchers who have been trained and 
socialized in the university Research environment* 

Additionally, because Evaluation Research is done in an op- 
erational, decision making context, the time lines may shift ~ \ 
i.e., decisionmakers may make "sudden requests" for informatita. 
Where the environmental context is public and political (as in ed- 
ucation), the probability that time lines may shift is even higher. 

For Researchers trained in Basic and Problem-Focused Research, 
such shifting times lines are likely to be a source of incredible 
tension — both because they would be unused to (and would dislike) 
3uch sudden demands and, because|,,^y findings reported under such 
conditions would, be incomplete and tentative — and could be in 
error. \ 

Thare is a clear implication here for an agency such as NIE. 
The needs of the decision makers and the resistances of Researchers 
both have validity. It would seem necessary, then, that some agency 
ensure that this tension is mediated, probably by building some form 
of preliminary/ interim reports into the Evaluation Reses^rch design. 

We must also note an inherent tension in most human service 
program Evaluation Research. Because the Evaluation Research is done 
in an operational context, the needs of decision makers will usually 
require the narrow time we discussed above. However , . human service 
programs involve "people change" — a change process that generally 
has a fairi.y long time frame, especially if community changes are also 
involved. That is, it may not be realistic to measure people change 
adequately or validly within th^e usually jhortcr time frames of Evalu- 
ation Research. 
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F» Tight Managfement 

In contrast^ to our analyses of the other functions. Evalu- 
ation Research appears^ to call for a rather tight monitoring a^d 
management l.e.» it li» critical that the Evaluation Research pro- 
cess be timed to the needs of decision makers. v 

• \ ■ . \ 

G. Modest Approaches ' \ 

The overarching purpose of Evaluation Research is the as- . 
slating of the decision m^lng process by reducing the level of 
uncertainty in decisionmaking* Further this assistance must be 
provided within a rela£.vely narrow time frame and must be speckle 
to a particular program. Thus, in contrast especially to Basic 
Research (but also, to a lesser degree) to Problem-Focused Re- 
search, SvAluaticn Research calls for a relatively modest approach 
-- I.e.. tbci ohj ctlve is to provide "sufficient information" 
rather than "eternal truth". 
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II. EVALUATION RESEARCH IN EDUCA'TION; A SOCIAL SCIENCE CONTEXT . 

Introduction; A Social Scie.nce Context 

The/Mstory and nature of Evaluation Research In the educational 
context[ if virtually Identical with the history and nature of Evalua- 
tion Research in the overall sociaSk science context. For that reason, 
ve will here focus our analysis on the social science context, refet- 
ring to tne educational context per se as relevant. 

1. Historical Development of the Evaluation Research Function 

Of all the R/D6el functions in the educatioa^ctor. Evaluation Re- 
search has experienced the most rapid and extensive development in the 
last ten to twelve years. 

Prior to the mid- '60*8, evaluation of educational programs (when 
it was done at all) was carried out by educational practitioners and 
by some Researchers — but rarely by people who identified them- 
selves as Evaluation Research specialists. The approaches tended to 
be normative, but rarely systematic or rigorous. The predominant 
strategy was casual observation and analysis. Conclusions tended to. 
be based on expert opinion, intuition', and impression rather than 
systematically gathered and rigorously analyzed empirical data. 

This pattern changed significantly in the '60s as large-scale 
federally-funded social programs proliferated, and the legislation 
that created them tended to riequlre the systematic gathe(^ing, analysis 
and reporting of empirical data on program effectiveness .\Thus, the 
Evaluation "Research function expanded rapidly as a new special 
even as a new industry: .in less than a decade, the dollar volume 
of federal Evaluatl<^ Research contracts expanded at least tenfold, 
vlth a sizeable portion of the funds being used for Evaluation Re- 
search of educational programs # 

2. A History of Methodological Issues 

During the •60s and early *70s,. there were many heated debates 
a3ong Evaluation and Research theorists about appropriate methodolo* 
gxes for the Evaluation Research function* One group argued that ex- 
perimental (or quasi-experimental) designs were more powerful than 
any other Research approaches for assessing the effectiveness of 
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prograsuy products, or str^^gles -^^ imd that it was therefore essen- 
tial to use these approaches to test outputs and social reform pro*- 
grams o£ all kinds. A second group argued that experimental approaches 
Ic^o&ed unrealistic constraints on field settings and that at any 

rate It could never be possible to meet adequately the statistical » 
design^ and treatment assumptions on which experimental approaches are 
premised. 

^ Other methodological debates revolved around the need £or evaluation 
approaches to provide feedback throughout the program development pro- 
cess not slnqply telling the Developer at the end of the Development 
process that his program did not work, but working with him throughout 
the process to make it better. Existing pre-post evaluation designs 
made it difficult for program Evaluators to provide this jklnd of feed-- 
back) or to understand how to evaluate a program stimulus that kept 
changing. 

Some of these disagreements have been eased by recognition among 
Evaluation Researchers that there are a number of different kinds of 
evaluation services , each requiring somewhat different approaches and 
tiBchnlques. The distinction between formative and sximmatlve .evaluations 
represents one such distinction. Initially, , the same Researchers con- 
ducted both formative and summatlve evaluations, but over time there 
appears to have been some specialization of personnel and organiza- 
tional units here. 

Currently, the formative evaluations that are undertaken as part of 
the R&D program/product development process are generally carried out 
by Evaluators who work with Developers as part of the Development team 
and provide ongoing feedback designed to Improve the prodpct^or program 
being developed. They use both quantitative data-based akd qualitative 
judgmental approaches. Their style of functioning emphasizes flexi- 
bility changing t;helr Research questions, variables, instruments, . 
and approaches as the emerging program takes shape and perhaps goes 
through a number of transformations. j 

The debate over experimental vs« other kinds of Research designs is 
now centered on sumraative evaluations the evaluations undertaken 

to test the effectiveness of a given program or product after it has 
been fully developed. Sut|mative evaluations are usually done by an 
evaluation agency or organizational unit independent of the program* s 
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Developers. Summatlve evaluations Include several types of evaluations 
dlf^ferlng somewhat In emphases because of the different information! 
needs of the decision makers to whom they are addrressed: .. 

1. final operational field tests of an R&D output to help the 
R/D&I'^manager determine whether or not it is ready for 
Dissemination; 

2. evaluations of the effectiveness of a given program or pro- 
iuct In a given school or District In, meeting 1 ally defined 

objectives; .1 
3^ evaluations of national prcgraai Initiatives, sampling prr*gram 
components nationwide to Inform federal policymakers about 
the effectiveness of a given strategy (or the relative effect- 
iveness of alternative strategies) In meeting federally de- 
fined policy goals. 
There Is still some disagreement about how appropriate experimen- 
tal designs may be for product tests and for individual school or 
school District program evaluations; and many other kinds of Research 
designs have been proposed for these types of evaluations. Nonethe- 
less, federal program evaluation policy (to whatever extent such a 
policy exists) appears to be moving toward experimental approaches -,- 
increasing numbers of national program evaluations are being conducted 
using experimental design.., control groups, and some randomization of 
treatments. However, the difference between experimental setting In 
the laboratory and the field Is gaining recognition. Federal Evalu- 
ators are increasingly acknowledging the need to supplement impact 
data with process data demonstrating that a given "treatment" was in 
fact implemented as specified in the program design, and that the 
impact evaluation is a valid test of the program and not siir ""y a 
"non-event," 

3. The Knowledge/Technology base 

The early phases of the maturation process of a knowledge and tech- 
nology base are illustrated with particular clarity in the enormous 
literature, produced by the Evaluation Research function over the last 
decade. Of all the functional R/D&I specialties. Evaluation Research 
appears to have experienced the most self-conscious and concerted 
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development; of its methodology during this period. The literature re- 
flects not only the Inherent difficulties of the Evaluation Research role 
and processes, but also the problems of weaning a new specialty away from 
a parent field. The earl- literature was filled with self-conscious 
analyses drawing distinctions between Evaluation Researcn and Basic/ 
Problem'Focused Research and emphasising the Inapproprlateness of pre- 
vailing Research methodology for the Educational Evaluation Research 
context. Within only a few years, the distinction from Baslc/Ptoblem- 
Focusad Research was taken for granted, and the literature documented 
\the development of Evaluation Research as a new field with a distinc- 
tive Identity. 

The rapid coming of age of the Evaluation Research function could 
be seen In the quick succession of seminal papers produced by Evalua- 
tion Research theorists, the publication of s;everal anthologies re«- 
printing Important articl(is oti Evaluation Research, the frequent cita- 
tion of the seminal papers of the field and the use of concepts, and 
approaches developed In these papers. It could be seen in the emergence 
of a somewhat common frame of reference among Evaluation Research theo- 
rists and a common vocabulary — Including such tenns as "formative" 
and "sumraatlve" Evaluation Research and "context," "input," "product," 
and "process" Evaluation Research. The maturation of the Evaluation 
Research function could he seen especially in the formulation of var- 
ious new Evaluation Research designs and methodologies, In attempts to 
develop taxonomies of Evaluation Research designs, and in the publica- 
tion of several handbooks synthesizing and compressing the accumulating 
knowledge and technology base and translating it into more readily usable 
reference form. 

Still, the conduct 6f educational Evaluation Research and the qual- 
Ity of Evaluation Research outputs have been the focus of considerable 
criticism. The field still lacks an adequate theoretical base. Eval- 
uation Research in^^truraentation is in a most rudimentary state of de- 
velopment. Basic conceptual and methodological dilemmas remain unre** 
solved. Though substantial progress has been made in recent years, the 
knowledge and technology base of the Evaluation Research function must 
still be considered relatively immature and underdeveloped. 
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4* The Study of Social Change; A Sqrlous iDlletnma 

The discussion of methodological Issues and of the knowledge/ 
technology base in Evaluation Research provides a context for con- 
sidering « serioua dilemna. S 

The study of social change is one of t^ie least developed sub- | 
Ject areas within the social sciences. Our understanding and measui^es 
of social change tend to be far less incisive and sensitive than they 
need to be, given the fact that Evaluation Research of social change 
can detenaine the fate of human service programs. 

Thus, the Evaluation itesearcher is faced with a serious diletma. 
He must assess change in a changing context (perhaps changing in 
part because of his presence) — but he must use methods which may 
be inadequately sensitive to the critical changes taking place. In- 
deed, he may not even be able to determine at the time which changes 
are fundamental and critical and which are Instead only fleeting and 
tangential -- i.e., he may be measuring and studying the wrong 
variables. , 

While to some extent this problem must be encountered by other 
Researchers in field settings, the problem is rarely as pervasive and 
central for other Researchers, who f.nd to be less critically con- 
cerned with detecting change and who may have less need to assess 
change in relation to a set of variables so" amorphous and changeable 
as the defining characteristics of a given human service "program." 

5> The Personnel/Institutional Base 

The discussion thus far has Indicated that there exists today the 
beginnings of a significant Evaluation Research community w.'.th its 
own separate identity. This fact is illustrated by the existence of 
journals, books, articles and even university training programs which 
focus specifically on Evaluation Research. 

However, the field is not yet mature and has certain character- 
istics which must be considered here. 

First, there are as yet relatively few persona well trained specif- 
ically in Evaluation Research, though many more than was the case 'In 
the mid *6Qs when the sudden upsurge of federal funding for Evaluation 
Research (coupled with much lower levels of funding foorBasic aud 
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Problem-Focused Research) created a vacuum that was rather natutally 
filled by Basic and Problem-Focused Researchers — Researcl^ers Who were 
. trained primarily in psychology and sociology. While these Resei^rchers 
do fill a gap in Evaluation Research, it jnust be noted that they iwould 
tend to be quite frustrated by the relatively formidable constraints 
that exist within the Evaluation Research function (e.g.! ti^e frame 
constraints; the lack of control over problem definition). Such' 
Researchers would also have the tendency to redefine Evaluation Research* 
problems and questions into Rasic/Problem-Focused Research types of 
problems and questions.. 

Secondlyi the bulk of federal Evaluation Research funding has gone 
to private sector organizations ai.d relatively little has gone' to the 
tinlverslty setting. In the Evaluation Research industry as a whole, 
statistics for HEW-funded Evaluation Research Indicate that in 1970 
45% of these federal funds went- to for-rprofit firms; 29% went to not- 
for-profit organizations; 21% went/to the universities; 4% went to State 

f 

and federal agencies; and 17% went to individual consultants (Abert, 1971). 

Rossi (1976) has suggested that naivete in Evaluation Research pro- 
curements has "turned off" the best Researchers. That is, Rossi sug- 
gests that while it is easy to do Evaljiation Research poorly and naively, 
good Researchers have recognized (and thus shunned) Evaluation Research 
procurements which pose questions that cannot be answered or cannot be 
answered within existing constraints. In a similar vein, evaluation of 
Evaluation Research by the Russell Sage Foundation indicates, that the 
high quality Evaluation Research tends more often to be done by the academic 
community than by the relatively new private sector Evaluation Research firms 
that make up the bulk of the Evaluation Research "indus:.i;y. " 

The existence of an Evaluation Research community (ev»::*n fnough not yet 
mature) and the evidence that quality Evaluation Research tends to come 
from the academic community would seem to imply the need to reconsider 
Evaluation Research funding policies in terms of selective support to 
facilitate the maturation of a core community of 'competent Evaluation 
Researchers. 
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6. A Value -Laden Polttlcal Context 

Evaluations are often describ'*'' as management tools designed to 
provide a "rational basis f^r decision making" but decision 

makers in the public sector function in a largely political sphere. 
This fact raises important issues for the Evaluator on both theoret- 
ical and practical levels. 

On the theoretical level, we must ask if political considerations 
are "Irrational," or are they based on "a. different moditl of ration- 
ality" from the one generally used by social scientists? 

On th^ practical level, consideration must be given t^ the poli- 
tics of decision making. Generally speaking, programs arq created 
by political coalitions of diverse interests — interests which 
support programs for diverse reasons. These coalitions tejad to view 
negative Evaluation Research findings unfavorably — ancjj generally 
have enough influence to modify or bury negative findings and keep 
their programs going regardless of what Evaluators report. Converse- 
ly, (yet similarly), programs may be opposed by other political inter- 
est groups — interest groups who will use findings of Evaluation 
Research to achieve their ends. Thus, Evaluation Research findings 
may be used, misused, modified, reinterpreted, buried, etc. -- in 
other words, used as a "political football." Given the political 
context and the methodological issues we have noted ^bove, it is not 
surprising that controversies over negative findings often fgcus on 
methodological rather than substantive issues raised by the findings 
themselves* ' 

Additionally, we must note t'hat the educational context is value- 
laden, and value choices enter virtually every <ine of the key deci- 
sions made by. the Evaluator The outcome of Evaluation Research may 
be predetermined by the choice of Research questions and objectives, 
the criteria used in judging effectiveness, and the measurement in- 
struments administered. From the human perspective, the question 
must be asked: Is the Evaluator value-free when doing Evaluation 
Research? From the organi2ational/political context perspective, 
the question must be asked: To what extent is/should these key 
value decision choices of the Evaluator be influenced by the organ- 
izational information needs of the decision maker on the* one hand 
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and the political context /dynamics on the other hand? 

In contrast, Basic and Problem-Focused Research are not so immersed 
in and vulnerable to a political value- laden context. They do nqt so 
directly affect the public or decision makers and they tend to be 
reiaote from the political arena. 

Th^ political value-laden context raises some important issues for 
the Evaluation Research function. Most importantly, it becomes critical 
that potential political issues be dealt with at the beginning of the 
design stage." This would imply a fair degree of Interaction between 
the decision makers on the one hand and the Evaluation Researchers. If 
political issues are not dealt with at the beginning , they will most 
likely have to be dealt with later in a highly dysfunctional '^attack/ 
defend'* context ^ and significant findings are likely to be muted, sup- 
. pressed or challenged on '^methodological'* grounds. 

A second important issue is that unless they are dealt with, poli-* 
tical issues^ are likely to be a strong source of frustration and ten** 
sion for Evaluation Researchers — a fact that can be highly dysfunc- 
tional to the long term health of the Evaluation Research function (and 
thus to the total educational R/D6el system). 
J A tfiird potential issue is that Evaluation Research may be used by 
program administrators (and perhaps even funders/policy makers) as a 
"court of last resort," a "panacea" to bail out and save a failing 
program. 

7. Control Over Outcome 

A very strong issue for Evaluation Research that is generally not 
found in Basic/Problem^Focused Research is the issue of who controls 
or "owns" the. outcome (the findings). This is both a theoretical/ 
definitional issue, a practical issue and a political issue. 

As a definitional/theoretical issue, waJnote that because Evaluation 
Reseat ^.:v Research, the implication would be that the public and the 
educational R/D&I system (through the Evaluation Researcher) should 
have accer.s to the findings of Evaluation Research. Additionally for 
govejpfllWnt- funded Evaluation Resaarclji, there would seem to be a reason- 
a^e presumption that the f:^ndings are public data* However, we must 
also note that since the purpose of E^'aluation Research Is to assist 
decision makers, it is they who initiate, procure and receive the find- 
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ings of Evaluation Research, we might assume that the decision mak&y 
has control over Evaluation Research Outcomes. 

However, there are practical matters to be considered. As t^he 
one; who initiates, procures", funds, receives and uses the Evaluation 
Research findings, it would seem reasonable to assume that decision 
makers have and will exercise control over outcome, especially given 
the political, value -laden context. Thus, for practical purjposes we 
must assume that the decision maker controls the findings and that the 
Evaluation Researcher has, in fact, no control over hbw the decisio^ 
maker will utilize the findings — a fact that can be a source of 
frustration and tension to the Evaluation Researcher. 

One additional aspect of this issue warrants consideration. The 
program administrator may have valid ^reasons for not wanting prelim- 
inary findings to be made public prior to the final summat^.ye Evalij---^ 
ation Research report. Specifically, the program^admiitiStrator may 
feel that in such a political, value-laden context, the release of 
preliminary findings (which may be in error) could lead to unneces- 
sary but seriously dysfunctional interventions in the life of the 
program beforie the program has had a chance to "mature." On the 
other hand, policy makers and funders may validly request prelimin- 
ary findings in order to monitor the program. 

The resolution of this issue would probably bt to have only lim- 
ited circulation of the preliminary findings, with the understanding 
that these would be available for use in the final evaluation report, 

8. The Educational Context 

While the educational context for Evaluation Research is basically 
the general social science context, there are some sectoral-specif-'c 
characteristics that v* should note, 

A, Design Variations 

Because of the size and variety within the educational sector (and 
because we do not yet have a fully mature theoretical and methodologi- 
cal base for social science Evaluation Research), consideration must 
often be given (for the larger-scale programs) to planned or unplanned 
variations in the programs which are being evaluated and thus poten- 
tially in Evaluation Research designs. An example of planned varia- 
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tion is the Follow Through program, which permitted local selection of 
model variations. 

Variation In programs and Evaluation Research design may be planned 
, •» or they may be unplanned. Given thab the educational system is a 
large, autonomous , decentralized and highly Varied system, there is 
really no way to ensure fuLly that actual implc&mentation will be the 
same as the desigrid (planned) implementation. 

Whether planned or unplanned, variations in program implementation 
and in Evaluation Research design create significant problems for 
con^aratlve analysis across settings and for aggregation of a data 
basig. , 

B> Multiple Levels of Governmen t 

Probably more than the other sectors » educational R/d&I Involves 
all levels of government » Inherent in this tact is a problematic ten- 
sion for Evaluation Research. On the one hand,^^cision makers at the 
various levels will have different information needs (including both 
practical and political information needs) • To meet these different 
information needs would likely require collection of more arid differ-* 
ent data than would be needed for decision makers at a single level of 
government. To meet these different information needs would quite pos*« 
sibly require a somewhat different (or at least a more ext^sive) Eval-* 
nation Research design — perhaps several separate designs. As might 
ha illustrated in the case of Tiule I, there is the danger of reducing 
Evaluation to the trivial level. 

On the other hand, the need of those who fund and make policy de- 
cisions from Evaluation Researv.h need^ cumulative , aggregatable data 
which is comparative across settings. There is some trend for govern- 
mental agencies to require '^^^rtain ^ata to be uniiarm (e.g.: New York; 
Office of 'Education)* 

The issue for Evaluation Rtsearch designing is thus the extent to 
which varying information needs can be met while still providing cumu- 
lative, aggregatable data which is comparative across settings » 

C. Awareness of Format ive/rummatlve Evaluation Research Issues 

We 1. ^ already noted the issues related to formative and summative 
Evaluation Research. There was a concern in education that the forrna- 
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tive Evaluation Research needs were not being met. The response to 
this concern has been for Evaluation Researchers to work with De- 
velopment teams in educational labs and for some evaluation consul- 
tants to work with school districts in order to provide feedback to 
them. Thus, the education field has developed a certain degree of 
awareness of formative/summative issues. 
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III. IMPLICATIONS FOR NIE 

1, Orchestration as the Major NXE Role 

As we have found true for th^ other functions we have analyzed in 
this study, the major role of' NXE in relatfon to Evaluation Research 
is that of system orchestration. In Evaluation Research, the orches- 
tration will be both complex and vital. The complexity is illustrated 
by the aspects of orchestration which are described below. However, 
probably the, major underlying consideration is that given the highly 
political and value-laden nature of the education sector, Evaluation 
Research must be done well — and it probably will not be done well 
without NXE leadership in orchestrating the various complexities in- 
volved. In this light, we npw focus our discussion on those aspects 
of tho Evaluation Research function which require orchestration. We 
now tUm to a consideration of thnse aspects. 

2, Foinnative and SuCTmati^''e jSvaluation Research 

The nee4 for orchestration of formative and su^rmatlvG Evaluation 
Research two- fold: 

.^1* Inhere Is a tension bptwe.en the need for "pure" data for sum- 
matlve Evaluation Rcsea; and the ^act that formative Eval- 
uation Research does Introduce change into programs. This 
tension .will need ^.o be mediated in the designing of the Eval- 
uation Research • ^ 

2. NIE wiU have to uak& decisions concerning the issue of 

whether a single Researcher can validly do both formative and 
summatlve Evaluation Research, or whether separate Evaluation 
Research organizations are required » If uhe latter course is 
chosen, then NIE will have to orchestrate the Researchers in 
terms both of the tension between the two ni^^des of Evaluation 
Research and the need to ensure that the data obtained from 
both Evaluation Research organizations is compatible and cumu- 
lative. 

3, Differing Information Needs 

Here> NIE will n^ed to determine for which levels of governmental 

\ 

agencies information needs are to be met* To the extent tliis includes 
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several levels, NIE will need to ensure that the Evaluation Research 
findings are cumulative, aggregatable, and comparable across set- 
tings and are not reduced to the level o£ triviality. 

4. Tha Political Context 

We have noted that Evaluation Research is done in a very politi- 
cal, value-laden context, with the result that Evaluation Research 
findings may be controversial and mayibe distorted, suppressed, re- 
jected, etc. Thus it becomes vital that political issues be dealt 
with in the initial design stage in order to develop and specify the 
way in v»nich negative Evaluation Research findings would be dealt 
with. 

This would seem to imply that there must be communication between 
Evaluation Researchers, program administrators and policy makers/ 
funders in the design process . This would on the one hand tend to 
make the Research Evaluators more sensitive to political realities 
and constraints and would on the other hand tend to make decision 
makers more aware of the danger that Evaluation Research findings 
might be p re-determined by the Evaluation Research design. Such 
forums of communication would force consideration of Evaluation Re- 
search design objectives and designs and would make the bases of 
decisions more explicit. 

5. Problem Definition 

Final decisions about problem definition in Evaluation Research 
are in the hands of those who procure and fund the Evaluation Re- 
search — not in the hands of the Evaluation i searcher. However, 
the Researcher may be In a position to make valuabls (and in some in- 
stances, essential) contributions to the definition of a problem. 
Thus the need is '.o ensure mediation of a variety of perspectives 
about the' problem definition for a specific Evaluation Research 
project. 

6. Are Eval VationxReBeareh Findings "Public" Data? 
' ^ - v 

There is an issue as the whether or not (and to what extent) 
findings are to be considered "public" data. This is a political/ 
value decision, and it is not our intent here to debate whether or 
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not government-funded Evalua '.on Research findings are to be considered 
"public" data. We do want to note that failure to resolve this issue at 
the beginning of the Evaluation Research process is likely to result in a 
heightened degree of p ">litical controversy after the Evaluation Research is 
compl€;:e«^. ^"V,,^ 

One fUrthar aspect of this issue warrants further consideration. While 
program administrators and policy makers/ f under s need to have preliminary, 
interim reports, widespread disseminftv ion of such preliminary findings could 
result in^^unctional interventions in the life of a program*^ This potential ., 
might be resoled by limiting the initial distribution of preliminary findings, 
but ensuring that they may be utilized in final reports which O^uld have wider 
distribution and availability. 

7. Interim Reporting 

We have noted both the need of decision makers for preliminary interim 
reports and the potential frustration of Researchers about providing highly 
tentative findings — especially when the demand, comes unexpectedly. Thus, the 
orchestration role of NIE is to mandate that decision makers have information 
when they need itj ensure that information is not demanded which it is not 
feasible to provide; ensure that demands are not made unexpectedly on 
Researchers; and monitor to ensure that the periodically needed information is 
prov'.ied the decisio^^kker . 

Additionally, it would seem important that Researchers understand the 
needs of decision makers for preliminary data on the one hand and that de- 
cision makers be aware on the other hand of problems caused when unexpected 
demands are made or when the information requested cannot feasibly be provided. 

Thus, it would seem that a process for providing preliminary interim 
reports should be built into the initial Evaluation Research design. 

8. Staging 

In the Evaluation Research function, there are several rather clearly 
defined stages which have different basic. requirements. 
A. The Design Stage 

The design st^age of Evaluation Research is critical. It is necessary 
at this stage to idetitify potential political issues and determine how these 
will be dealt with-^ whJn the Evaluation Research i^ completed. It is necessary 
to insure that problem definitions and Evaluation Research objectives are clear, 
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relevant and within the limits of Research feasibility — and that a variety - 
of perspectives are considered in determining the problem definition and the 
Evaluation Research objectives. It is necessary at the design stage to de- 
termine whose information needs will be met and to determine what interim 
reporting is to be provided. 

Because of the critical nature of the design stage, it is vital that 
the b^st Evaluation Research -talent be obtained — a fact which would suggest 
hat NIE must exercise a relatively high degree of control over the selection 
pnocess. 

B. The Data Collection Stage 

Once the problem definition and the Evaluation Research objectives are 
clear, there getjierally would be a whole range of personnel and organizations 
competent to perform data collection. The "best minds" are not necessarily 
needed. Thus, a competitive bidding mode would be refevant. 

There is an exception. In the instance where there is a high level of 
uncertainty about how to define the problem, what methods are really valid, 
etc., the Evaluation Researcher must have a high level of understanding of 
the area to be researched and must be creative and innovative in designing 
the Evaluation Research. Xn this instance, particular skills are required; 
thus, NIE control over the se,lection process would be a more relevant strategy 
than open bidding. , , \ . 

C. The Data Analysis and Reanalysis Stages 

The findings of Evaluation Research are critical both in the sense that 
important policy/funding decisions will be based on the findings and in the 
sense that education is a politically sensitive, value-laden area. It follows, 
then, that it is important to obtain the strong analytic personnel of the field 
for data analysis and reanalysis. In this context, NIE control over the 
selection process would be the relevant strategy. 

y For th'' same reasons, data reanalysis is often provided (e,g,: the 
Coleman Report; the Racial Isolation Report ). Thus, a strategy might be con- 
sidered whereby more than one Evaluation Research organization provides 
simultaneous data analysis. In effect, this would be doing data reanalysis 
during the basic data analysis stage. 
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D, Formative and Sunmative Evaluation Research 

Formative and summative ; valuation Research require orchestration at 
two points* Firsts the tension between formative and summative Evaluation 
Research must be mediated. Second, decisions must be made as to whether 
formative and sxmmiative Evaluation Research will be done by the same or. dif- 
ferent Evalxiatlon Researchers. If done by different Researchers, their efforts 



gregatabl^ and coiq)arable. 

«■ 9. Portfolio Effects on System Capability Building 

Because there are various Evaluation Research organizations already in 
the field, system building per se would not appear to be a primary concern 
here. However, we have noted that while the best Evaluation Research talent 
and the best Evaluation Research work tends to be found in the academic setting 
federal funding is far more supportive of private sector entrepreneurial organ- 
izations. This would imply a systein capability mode of funding which selec- 
tively fdcuses on the academic sector, and possibly even focuses upon a specific 
set of imiversities and university-related organizations. 

10. Monitoring and Tight Management 

The emphasis in Evaluation Research is that it must be done well . Thus 
the process must be closely monitored. A "tight management" mode would seem 
to be relevant Jn order to insure that basic requirements are met — yet care f 
must be taken to insure that the management is not unnecessarily and unduly 
restrictive. 

11. NIE Internal Considerations 

A. Relation of NIE to the Field . 



appear to require that NIE actually be Involved In Evaluation Research. Thus, 
we have suggested, the need appears to be for NIE to have top level personnel 
who have a combination of substantive/methodological skills and political savvy. 

B. NIE Personnel 

As we have noted, NIE must have some top level personnel who have sub- 
stantive and methodological Evaluation Research skills, on the one hand, and ^ 



must be coordinated to insure that the data of both will be cumulative, ag- 



The orchestration requirements of the Evaluation Research function 
suggest the need for NIE to have Dngoing, close working r^ationshlp^ with 
the field, but do not (in contrast to Basic and Problem-Focused Research) 





) 

• . \ 

political savvy and insight, on the other. Such persons would be able to 
understand the critical implications of: different ways of defining problems; 
different Research methodologies and their televance to types of programs and 
program situations; and the needs of the different pkrticipants — i.e., the 
policyoakers/f under s, the program administrators, aAd the Evaluation Researchers 
Additionally, it is imperative that NIE personnel have skills relevant to the 
process of orchestration. 
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SUMMARY ANALYSIS ACROSS FUNCTIONS AND CONTEXT; 
IMPLICATIONS FOR THE ROLE OF NIE 

« 

• mRODUCTION 

For a federal agency such as NIE, the value of an analysis of R/D&I 
systems is detHrmined Ky the implications that can be derived. for policy 
and sti^tegy decisloi]LS i.e.,* the identification and evaluatic f 
policy/ strategy options and issues. At this point, however, it becomes 
Ijiperative to place some practical limitatims upon the nature and 
extent of "R/D&I system analyses. 

On the one hand, time and cost ronsiderations make it im^^ossible to 
undertake a "theoretically complete" ar ^ysis — tb? considerations 
involved in a total contextual analy of all functions and functional 
issues would be horrendously massive and coaplex; the needs of the 
educational R/D&I system rrr more thna N''fi could possible respond toj 
there is a vast array of possible options available which are potentially 
relevant to NIE purposes. Thus 5, the analysis of the system must be 
narrowed down to a feasible scale. 

On the other hand, the analysis must remain sufficiently broad and 
rich so as to provide insight and guidelines for policy and strategy 
decis^ions. 

Agency personnel need not be overwhelmed by tho number and di- 
versity of unfamiliar courses of action suggested. Rather, NIE should 
accept the fact that it cannot do all the kinds of things we have sug- 
gested, at least in the shore run. T . Agency will be selective (as it 
shovld be) and determine its priorities. 

To provide a framework within which the Agency can identify and. eval- 
uate relevant policy and strategy issues and options, it is essential for 
the Agency to have a basic understanding of: 

1» the R/D&I system £s £ system — a system to be understood in 
terms of a otal, interactive context which will include the 
system' u environment, the operative conditions existing with- 
in the system, and the various system functions; 
2. generic and sector-specific system characteristics; 
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3* similarities and dissimilarities between the various system 
functions; 

4. the implications o£ the kbove for determining appropriate 
Agejacy policies and strategies « 

The analyses of the four functions 5>rill have begun to provide such 
a framework for the Agency* We now turn to a cross-function comparative 
analysis (in summary form) to add to t^he Agency's analytical/decisional 
framework. j 

!• A CO>fPARATIVE ANALYSIS ACROSS FUNCTIONS: DEVELOPING A SYSTEM CONTEXT 
OVERVIEW / 
A cross-^functional analysis suggests that the overall requirements 
for NIE policies and strategies arise out of three bases: 

1. a sense of NIE*s mission as one -involving system building and 

various subtle forms of sjrstem management; 
2* an understanding of the general immaturity of the 'educational 
R/D&I System at the present time, what this suggests about the 
appropriate roles to^be played by a lead agency w*^ th responsi- 
bility for building an ''effective R&D system" and w;hat changes 
in Agency behavior might be died for over time as the system 
matures and the 'capabilitiies of the field are better developed; 
and 

3« the political, value-laden, social science based nature of ed- 
ucation. 

We shall consider each of these points in turn. 

\ 

\ 

U The Mission of NIE 

Before any ^consideration can be given to specific policies and 
strategies^ we imist addres th& more fundamental issue of how NIE 
sees its diission. This issue may also be defined from another per- 
spective. By definition, MIH is a funding agency of the federal 
govermaent. Thus, the key question becomes: What "mission'^ per- 
spective will determine how NIE allocates the (limited) federal 
funds under its control? The answer NIE gives to this question will 
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largely determine: 

1. what NIE will and will not do; 

2. what effects "NIE will and will not have on the tota] edu- 
cational system in general and on the edt rational R/D&I 
system in particular, 

A. The NIE Mission: Narrow or Broad 

In the most general sense ^ NIE could define its mission from 
either of two perspectives: 

1. A Narrowly Defined Mission: A Passive Channel for Funding 
\ NIE could define its mission narrowly as being simply a 

passive channel of funds from the federal government to 
the education sector. Such a narrow definition of mission 
V Ight seem appealing at first glance, but it provides no 
real basis for making choices of allocation of limited 
resources among multitudinous projects, educational organ- 
izations, etc. At best, this perspective would permit some 
kind of simplified "percentage distribution" formula (e.g.: 
10% to Research; 10% to Development; etc^i or 25% to SEA; 
etc.) At worst it ignores the full implication of the need 
to have the appropriate quality institutions and personnel 
that should be utilizing these funds. 

2. A Broadly Defined Mission; System Oriented Responsibilities 
On the other hand, NIE could see its mission more broadly; 
i.e., accepting the responsibility to also uce federal funds 
to facilitate and develop the educational R/D&I system. This 
system-oriented mission would include coacern for system 
building, system maintenance, system monitoring, etc. as well 
as the channelling of funds for direct product procurement 
purposes in a word, a ^-esponsibilicy for determining and 
sha'^ing the nature and direction o^ the r'ystem. From this 
nissional perspective, NIE would have the key role of using 
its funding capabilities to orchestrate the various parts of 
the system* 

As the discussion in the previous sections of this report indicate 
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CISST is assuming that NIE has and accepts the broader, system-oriented 
definition of its mission* Our assumptions are derived from our knowl*- 
edge and study of K/D&I systems in general, our discussions and ex-* 
p.erience with NIE personnel, and our analysis of the R/D&I context of 
the education sector in particular* 



stitute is likely, to disagree with our nystems analyses. More importantly, 
however, the educational R/D6el system is likely to remain immature, under- 
developed, weak and ineffective — with likely long-term negative effects 
for the Institute and the whole educational R/D&I system. Given the nature 
of R/O&I systems in general » and weaknesses of immature R/D&I systems 
(such as education) in particular, some system-xd.de agency must perform 
a system-oriented role. In the education sector, NIE as the lead agency 
for Research and Development appears to be the most likely candidate and^ 
probably the only agency with any inclination toward performing this role* 

1l NIE does accept a system-oriented mission, then this must be 
made clear to all Agency personnel, to members of the R/D&I and oper- 
ating system, to Congress, and to the public as well» 
2> The State of the R/P&I System 

If NIE accepts a system-oriented mission, then this\mission must be * 
accomplished within the context and set of need^, oppox^tunities and con- 
straints dictated by the state of development of the educational R/D&I 
system* Effective strategies tend to be,/context-bound, appropriate to 
the reality of a given set of contextual conditions in a given time and 
place. Thus, policies and strategies cannot be based on abstrations: 
e»g,: that field-initiated R/D6el activity is always "good*'; or that 
Agency directiveness i** either always "necessary" or always "oad"» 

A. Mature and Immatuxe Systems 

Broadly speaking, R/D&I systems may be described as being relatively 
mature or relatively immature. 

For example, in a mature system, one would tend to find: special- 
ization among functions; a basic, solid core of trained and experienced 
personnel within each function; communication networks which facilitate 



B, Some Implications ^^*^^>>^^ 
If NIE does not accept; this system-oriented mission, then the In- 
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information flpw within and between functions (e.g.; journals, "In- 
visible colleges"); Users 'who can differentiate between "good" and 
"bad" products; Users who have (or can obtain) the technical capa- 
bilities neqessary for successial Implementation/Utilization of in- 
novations; etc. y etc. 

In an immature system, one would tend to find the above to be 
lacking l.e», there would be numerous critical "gaps" in the system. 

Somo Illustrative Implications 

To /Illustrate the nature and implications of the differences 
between mature ^d inmature systems, let us look at the following 
examples: _ ^ ' 

1. System Building 

In an immature system, the weaknesses and^^aps of the system 
indicate that system-building needs to be a major policy/ strategy 
emphasis. Inya mature system, the major aspects of the system already 
exist and are generally functioning well* Thus, in terms of capacity, 
system facilitation rattrer than syst^ building becomes the relevant 
policy/strategy emphasis* 

2. The Dissemination Function 

As another example, in an Immature system, there is a need 
to "fill gaps" and to develop Dissemination mechanisms. In a mature 
system, one would work with and through existing Dissemination 
mechanisms. Further, we should note^M:hat some of the roles and 
mechanisms which would ueed to ue created^ta .aexve the "needa» of an 
immature syste *ould not be needed (and could even became dys- 
functional) z the system matures. Thus, it may be necessary to 
build in change (and even terminatioa) for thoae Dissemination 
mechanisms which are d veloped within and for the Immature system, 

3. Ager::y Dlrectiveness Toward the Field 

An immature svstem will tend to lack self-controlling and self- 
orchestrating (organizing) capabilities. Thus, it is likely that the 
Agency will have to provide a significant amount of direction (though 
how this is done will x'ary across the functions). In a mature system, 
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less Agency directlv«i\ess will tend to be needed because self-controlling 
and self-orchastrating mechanisms will exist within the field (e.g»: 
through ehte "Invislbtle collegea"; through quality control mechanisnis 
built into the Dc^velopment process). 

C. The State of the Educational R/D&I Syatem 

The fuiictioaal analyses of this study reveal two basic characteristics 
of the educational R/D&I system* 
It Varied and Large 

The User population in edi^cauxon io quite varied. The User may be 
seen as teachers, administrators, local and State agencies, and (ultimately) 
students. School Dist icts will vary by size,, level of funding, pro-* 
fesaionalism of teachers, needs (e.g.: urban /rural": , cstc. Within a 
single school, there may be varied needs (e.g.: vocational education; 
bi-lingual needs; etc.). The publics of the educational environment are 
also many and varied (e.g.: parents; political groups; etc.). Further, 
the potential Users are numerous thousands in terms of school districts 
alon*^ 

2. Immature 

The educational R/D&I system is clearly immature. There are sig- 
nificant gaps in functional specialization. Both the institutional and rson- 
helbase tend to be weak* Many of the existing institutions are weak 
or are inappropriately organized for carrying out certain functions. 
Some may have been. hurt by previous federal funding policies that shifted 
their nature from organizations carrying out the full range of R/D6el 
functions to more na^Towly specialized Development organizations. The 
field has attracted an inadequate supply of first rate talent and much 
of the work that is produced is poor in quality. The field's knowledge/ 
technology base is weak. Communication mechanisms for information flow 
and quality control are underdeveloped. There is little consensus on 
standards for judging the quality of outputs. User system capabilities 
for selecting, adapting, and implementing externally developed innovations 
are weak. In all, then, the educational R/D&I system is immature and 
requires Agency behaviors appropriate to an R/d&I system in an immature, 
state of dev€(lopment. 
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3> Tha Nature of Education 

Education as a sector Is highly vulnerable to social and political 
Influence and is often subjected to considerable pressure from the 
public and fron Cdngress as well as from the varied partlc^lpants with- 
in the R&D and operating systems. Schools are public service insti- 
tutions supported by public funds and regulated by public agencies. 
Education by its nature tends to have diffuse goalsN^at are subject 
to value judgments, misinterpretations and controversj\ — goals that 
are harder to specify, less measurable ^ and harder to use as standards 
against which to judge system performance. Contributing to this vul- 
nerability is the weakness and uncertainty of the field's knowledge/ 
technology base and the public's view of itself as having miJich knowlr 
edge about education (in contrast to such other fields as health) • 

Given the dependence of both the operating system and the R/D&I 
system on public funding, and the generally negative elimate that 
has surrounded R/D&I funding in recent years, substantial clarity 
about the Agency's long-term system building role, and its implications 
for Agency procurements an4. other actions, would seem to be essential. 

In sum, we have suggested that Agency strategies must be developed 
with a view toward system-orlente4 responsibilities conceived in terms 
of the state of maturity of the educational R/D5el system and the 
vulnerable nature of the educational sector in general* 

In the following sections we will be summarizing key :tssues and 
implications across the functions we have analyzed. The issues \*e 
have already fosed will, inevitably, reappear - each in their appropriate 
functional context, giving the appearance^ of some redunda xy. This 
is inevitable- and proper given our policy making ^rms --ia focus Lbat 
often finds redundancy preferable to elegance and parsimony. 

II, A COMPARATIVE ANALYSIS ACROSS FUNCTIONS; 'THREE MAJOR COMMON TIIEME S 

With the above overview perspective in mind, we now turn to a 
comparative analysis of the four key R/D&I functions which we have 
previously analv^ed separately. 

It is important to note that a number of common themes appear 
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across all four functions. At the same time we must note that the 

operatlonalizatloa of .each of these themes tends to be different with- 
in each function* It is important for NJE persi n.nel to be sensitive to 
these differences* The differences are substantial -in terms of some 
of the&v^y less significaat in relation to others. We now examine each 
of these common the^nes in relation to each of the four functions. 

Three of the common themes are of sufficient significance to war- 
rant separate attention at the beginning of our cross-functional analysis: 

1. A requirement for NIE leadership; 

2. System building; 

3. Orchestration as the major NIE role. 
I. A Requirement for NIE Leadership 

A. The Need for NIE Leadership 



Given the size, variability and i^mnaturity of the R/ D&I system, 
there is clearly a need for a nationwide agency to exercise a system ^i, 
leadership role and to provide direction for the system. NIE is the ^ 
only agency likely to undertake such a role in the education sector* 

B> The Form of NIE Leadership 

The need for NIE leadership must thus be taken as a "given" • For 
policy and strategy purposes, the issue now becomes "*he form of NIE 
leadership. We have noted for each function that the nature of the 
function and the state of the educational R/D&I system indicate that a 
process mode of management will likely be more effective than a management 
mode based on administration of detailed plans. Even here, huwever, the 
form of process management will dif^^ somewhat across functions (as we 
shall discuss later). 

C. The Agency vs. Field I&sue 

This finding has a specific iraplicatio..; for the Agency/Fi^Id 
relationship issue. That is, the relevciit issue is not so much o. 
of Field- Initiation vs. Agency-Dlrcctiveness as J t is one of the way 
in which NIE will pro\lde the leadership which only It can provide* 

D> Fun ctional Areas Differences 

Baste Research 

Given the high level of uncertainty .involved in Basic 
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Research, the NIE leadership role will be one of selecting 
Research areas and identifying quality activities in the 
field to support through NIE funding* 

Problem-Focused Research 

In Problem-Focused Research the key decisions are likely 
to be those involving selection of strong institutions for 
long-term support in an institution building (or re-building) 
mode» 

Development 

In relation to large-scale Development organizations^ < 
the needed NIE leadership role is likely to be on the aiia^ly 
stages of Need Identification, i.e., providing some direction 
to these organizations about the kinds of Development out- 
puts needed. In relation to practice-based and practice- 
related Development, NIE leadership will be required to buy 
and/or rent neaded capabilities to strengthen this Develop- 
ment mode. 

Dissemination 

As we have noted. Dissemination is a system-creating 
function and is thus vital for R/D&I system building efforts. 
Since the Dissemination function is relatively weak in ed- 
ucation, NIE will have provide leadership in the designing 
and building of the educational Dissemination system. 

Evaluation Research 

Two aspects of the nature of Evaluation Research in 
education require that NIE closely monitor end orchestrate 
the Evaluation Research function. 

1. NIE must provide leadership to ensure that potential 
political issues are resolved (as far as possible) at the 
design stage of Evaluation Research. Political issues which 
are not dealt with until after the Evaluation Research is 
completed may very likely to destroy any benefits that might 



have been obtained from the evaluation. 

2. NIE must provide leadership to mediate the varying 
Information needs of different system participants on the one 
hand (i.e., program managers, funders, and evaluators) » and to 
ensure on the other hand that the/ Evaluation Research I9 not 
reduced to a meaningless 'loweat common denominator 

2. System Building 

A. The Need and Implications 

Simply because of the relatively immature state of the educational 
R/D&t system, we have noted in each function that system building Is a 
key long-term need. This implies that system building should be one of 
the main criteria for NIE policy/strategy decisions (and thus, also for 
project selection, etc.). This further implies giving consideration to 
multi-purpose and portfolio effects in project and contjtractor selections. 

s^he functional analyses also rioted that organizations and roles created 
to meeK the needs of an immature system may be unnecessary and even dys- ( 
functlowir in a mature system. Since organizations tend toward self- \^ 
perpetuation and self-expansion, system-building must include the capacity 
to respond to change even to termination of some aspects of the system. 

Finally, the functional analyses noted that in an immature system, 
the kind of system building that is needed in the long term may not be 
what system members want at the present point in time. 

B. The Field vs» Agency Issue 

The above analyses further illuminate the Field/Agency issue. Speci- 
fically in relation to system building, NIE will have to assume a major 
responsibility/ 

C. Functional Area Differences 
Research 

We have stressed that the system-building process in Research 
is a slow process. Further, the rate at which Research systems 
can be built is limited by the extent of already existing centers 
of excellence. Thus, if more funding is provided than the existing 



216 



personnel/ institutional base of excellence can productively 
absorb, the result would tend to be mediocre Research and 
the laedipcre training of personnel. 

. The field has a strong role to play here in working 
with the Agency to guide decisions on which areas should be 
supported, which institutions, etc. At this point in time, 
the system-building purposes of any Research procurement are 
likely to have long-term implications as significant^ as (and 
often nore than) the specific substantive output of the Research 
project itself. The Agency needs to be clear and explicit 
about this» and develop some understanding for this position 
within the Research community, the Congress, and, perhaps the 
public more broadly. 

With regard to Problem-Focused Research, a re-building 
mode may be required to restore the Problem -Focused Research 
'Emphasis to many of the larger scale organizations that have 
been turned into Development organizations through the shifting 
priorities of federal funding in the late '60s, To increase 
the viability of universitv.^ettings as centers for large- 
scale, high quality long-term Problem-Focused Research, one 
option may be to create joint university-non-university con- 
texts for such Researchers, 

DeveloDPeat 

'* I^IE's system building, responsibilities in relation to large, 
-specialized Development organizations would seem to be to en- 
sure the development of mechanisms that lim: these organizations 

to Users. In relation to practice-based/practice-related De- 
ft 

velopment work, NIE will have to make decisions about buying 
and/ or. renting the needed capabilities to identify, package, 
produce and disseminate exemplary programs and practices. 

Dissemination % 

Since Dissemination (as a linking function) is system- 
creati^, system-building takes on special importance within 
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the Dissemination function; Given the complexities of the Dis* 
sealnatlon function and the need to permit the development of 
a redundant full-scale kind of system that provides Users with 
alternative channels of access to the resource base of the fields 
system design is the key to NIE's role* Given the immaturity of 
the overall R/D&I system, intermediary organizations will be needed 
between R&D Producers/ Suppliers and Users but such organizations 
must be designed in a manner that will mandate their ''withering 
away" as the system matures and makes them not only leas necessary but 
also somewhat dysfunctional. 

It is vital to note that the irate at which the system can 
be estajillsh^d and expanded must be congruent with the much slower 
rate c^jUser absorption of new information and new information 
sources^ Otherwise, the system will create unrealistically high 
expectations of rapid and widescale impact, thereby leading to 
disappointments that will have long-^term negative effects on the 
system* 

All of these requirements suggest the considerable complexity 
involved in this kind of system building and the need for great 
Agency skill in carrying out this task^ 

Evaluation Research 

We have noted that there exists the beginnings of an extensive 
Evaluation Research community with, however, a good deal of var- 
iability among the institutional and personnel bases* We have 
further noted that different stages of Evaluation Research re- 
quire different skills that are likely to be found best developed 
in different organizations. 

Thus, the system building role that Is needed to develop 
the Evaluation Research function is not one of system creation 
as much as Identifying, developing and orchestrating the existing 

Evaluation Research community* Critical will be the need to select 

\ 

tho^e relevant and qualified institutions upon which to build and 

» 

whose efforts should be facilitated* NlE*s own quality control and 
selection skills will be critical here* 
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We must also note that the political, value-iaden nature 
of educatloaal Evaluation Research underscores the importance 
. . of having a highly competent Evaluation Research community — 

and thus the importance of system-facilitation within the function. 

-3. Orchestration as the Major NIE Rple 

•• - ' ~ r 

A. Tne Need foi System Orchestratiot^ 

The functional analyses have indicate^ from a number of perspec- 
tives, the need for a system-wide and syspem-oriented agency (i.e., 
NIE) to provide orchestration for the educational R/D&I system as 
a whole. Such orchestration is needed because of: the system's 
immaturity; its variability; the differences in perspective among 
system members (e.g., Users vs. Researchers).; the Interdisciplinary 
nature of education; the need to balance strategies in relation to 
the Dissemination function. 

Given the need for orchestration, system'^ monitoring becomes a 
key aspect of the orchestration process. 

1 
I 

h» Th e Agency vs. Field^ Issue \ 

When the primary role of NIE is seen as sj^stem orchestration, 
the Agency vs. Field issue chaxiges (or even lopes much of) its 
meaning. Given that NIE must provide a systemtorlentatlon leadership 
with a key emphasis on system building, and gi^^en our understandings 
of the various system functions (as discussed in previous chapters), 
we can see that the real issue is simply who can do what best at 
what given stage in the development of the total, system and its 
functions, and bearing in mind the long term sys'jtem building needs. 
This is often a fine, dileirana, one that calls for subtle leadership. 
Thus, orchestration of both its own and field roles becomes the form 
of- NIE leadership. 

C. Functional Area nfferences 

Basic Research 

In Basic Research, Agency orchestration entaUs: 
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!• selecting Research areas from among ffll the possible Re- 
search areas that might be funded ('(•e*, "placing bets"); 

2« finding and supporting quality activities going on in the 
field (especially the work carried out in centers of ex- 
cellence possessing the necessary minimum critical masses 
of talent) » 

Such orchejtratioo can be carried out ef fectivelyTby^^^n agency only 
if it Is staffe^i by competent Researchers who are activ^ely involved in 
the conduct of Research, and are sensitive to shifts in the field and 
emergence of new-Hesearch areas » Basic Research cannot be orchestrated 
and monitored through tight specifications « Rather , it requires the 
presence within the Agency of researchers who are an integral part of 
the field* Advisory panels can be used but cannot be relied on alone • 
Given this close Agency/Field relationship and the active involvement 
of researchers in the field, the Field-Initiated vs. Agency-Directed 
issue tends to melt aWay» The role of NIK becomes one of selecting 
from a field in which the Agency is closely integrated. 

Problem-Focused Research 

Agency orchestration in Problem-Focused Research involves: 
1» mediating the tensions between the Research and Development 
perspectives, between Researchers on the one hand and 
Funders and Users on the other, and between the perspectives 
of the different disciplines^, 

2. orchestrating problem selection in terns of the oft-conflict- 
ing criteria of User needs vs* researcjiability; . 

3. orchestrating the types of institutional settings ijit which 
the Research is carried out (e.g., large scale settings in 
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the private, quasi-public and academic sectors); 
monitoring the Research proci 
problem focus is maintained. 



4. monitoring the Research process to make certain tluiu the • 



As in Basic Research, monitoring of Light output "^eeifications would 
be an inappropriate form of orchestration^ 

Orchestration for the Development function is a very complex 
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matter, bozi. because of complexities within the Development function 
Itself au.d recease Development activities must be orchestrated closely 
with other fractions. 

NIE zs^BZ orchestrate a number of system linkages: Development/ 
Production "-'rV-^gps; linkages between the Developer and the state of 
the art; llikagas between a variety of Development institutions; 
linkages to iosare. User input throughout the Development process, j 

NI]^ Exi^z orchestrate the proper roies of NIE and the field in 
Development; i.e., insuring field input in project selection and 
evaluation, crcbestra ting the selection and placement of a4viser5, 
emphasizing 3ZE responsibility in project and system orchestrai^ion. 
Project selsczica in particular requires an NIE orchestration rol^ 
simply becanss there are so many needs and possible projects r«* i.e., 
more needs r'-^ can possibly be met and thus more potential projects 
than can be funded. Further, the variety of needs indicates a con- 
sensus builii:^ need, which only Nifi could orchestrate. 

Durin? zhe Development process itself, NIE must orchestrate the 
various Su^=3 in the process and monitot for quality control through- 
out the process. 

NIF. o--st also orchestrate a balance between the various modes of 
Development; i.e., the specialized Development organizations vs the 
practice-bassd ts. the technological-opportunity-based modes. 

FinaLljs KI2 itself must know what does and doas not exist in 
the field and vnat NIE itself can and cannot do effectively. 

Dlssg^-"='tion 

Cr:bestration of the Dissemination function is also very 
complex, buz has the distinctive characteristic of having a User focus. 

In the firsz place, orchestration of Dissemination will be complex 
simply becs%-5e of the nature of the Dissemination function. Because 
Dissemination is by definition a linkage process s, NIE must orchestrate 
the way the Disseaination function informs Users, enables Users to 
utilize inzc-ratioas, and enables Developers to know x^hat Users need. 
Because Diss^inination is a systems-creating phenomenon, orchestration 
of R/D&I svszen designing becomes a part of orchestrating the Dis- 



221 



semination function. Finally, NXE must orchestrate the transforms between 
Dissemination and the other functions. -^^ 

Secondly, orchestration of Dissemination will be complex because of 
the variety of possible and desirable Dissemination mechanisms. Not only 
is there no single "right" Dissemination mechanism; a variety of Dis- 
semination mechanisms is needed to provide "fail-safe" for the system 
itself. Dissemination strategies will likely consist of a combination 
of Dis semination mechanisms and methods. There will be a variety of 
intermediary organizations Involved in Diasemlnation. Finally^ there 
must be a "fit*' between the "what" (products) and the "how" (Dissemina- 
* tion mechanisms). 

Similarly^ the size and variety of the User market will complicate 
orchestration. Further^ orchestration of Dissemination must allow for 

Dissemination to be initiated either by the Use; or by the Disseminator. 

ft 

Th^re are some mandatable features within the Dissemination function 
(for example^ allocation of resources to their markets) which must be 
orchestrated with the variety of more naturally energizing Dissemination 
mechanisms. The natural complexity of the function knd the need for a 
variety of Dissemination mechanisms and approaches will require an on** 
going NIE monitoring role. 

Evaluation Research 

Orchestration of the Evaluation Research function has three 

Icey characteristics; It will be very complex; it must be done in a 

\^jj^ue-laden political context; it must be done well. 

Orchestration of the Evaluation Research is complicated first be-* 
cause there are many different potential Users and participants. Program 
administrators » Funders, Evaluation Researchers and the public will all 
have different data needs and Interests — thereby raf.slng the issue 
and problem of data aggregation. l 

Further, there are three possible uses or purposes of Evaluation 
Research: 

!• formative **--r-i«e«, to provide project management with feed- 
back to help control a project while it is still in process; 
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2, summative -r i.e., data upon Xvhich the policymaker can base 
future decisions. 

3. knowledge production — i.e., adding to the data base of 
the educational sector and the R/D&I system. 

The above considerations raise a further issue for orchestration 
of Evaluation Research — i.e., control over outcomes. This is a 
political issue in the sense that various persons may attempt to 
reject, modify or supjJress findings; or may seek to use findings 
"out of context" for political purposes. Control over outcomes is 
also both A practical and a theoretical issue. The project manager 
may need feedback during the life of a project, but: 

1. the Researcher may not wish to provide findings which are 
only tentative and may be in error; 

2. feedback data may change the process and therefore make 
impact Evaluation Research difficult at best; 

3. The manager will likely not want the feedback data to be 
made public for fear that it may instigate dysfunctional 
interventions in the project before the project has "matured." 

The rbove oiscussion leads to three kay orchestration needs: 

1. orchestration to resolve potential political issues at the 
design stage of Evalu^ii;i:6n Research to the extent feasible; 

2. orchestration of control- over outcomes to attempt to mini*^ • 
mize.feedbaek made -*^Elu'~lio during the life of a project 

but that both feedback data and final report data will be- 
come public after the conclusion of the project; ^ 

3. orchestration of the tension between the need of summative 
Evaluation Research or "pure" data and the fact that the 
feedback process of formative Evaluation Research introduces 
process changes which make "pure" impact Evaluation Research 
difficult at best. 

From this discussion, we must conclude that the dema'nds on Eval- 
uation Research are great. Monitoring wilt-^De a key part of Evaluation 
Research orchestration. 

As if this were not enough. Evaluation Research orchestration 

must also include: 



1, the differeot types of Evaluation Research skills needed for 
standardized as contrasted to highly innovative programs; 

t 

2, the different stages of Evaluation Research which require 
different types of Evaluation Research skills and which thus 
may be done by different organizations; 

3, the Issue of having the person who does the operational Eval- 
uation Research (^txA may therefore no longer be objective" 
alM the impact Evaluation Research (anC If not, as i^ often 
so, tQ orchestrate the two sets of Evaluation Research) . 

III. A COMPARATIVE ANALYSIS ACROSS FUNCTIONS; SOME OTHER COMM ON TffiMES 

1. Varied But Close Agencv/Field Relationship 

We have tended to stress close Agency/Field relationships through- 
out our analysts. Illustrative of the range of relationships we 
see as possible are those that would entail such Agency actions as 
Initiating R/D&I activities, or coordinating, mediating, facilitating, 
supplementing, or evaluating R/D6el actlvitie's already operative in 

the field. ^ 

There is, of course, at least one more option — that of man- 
dating where the field will not or cannot do something that is 
needed. Even here, the Agency would need to have a fairly close 
working relationship with the field to be in a position to knew 
what the field will not or cannot do. This Agency/Field relation- 
ship Vill vary across the functions. 

Research 

In Research, the relationship between Agency and field needs 
to be closest of all functions. It calls for involvement of Agency 
personnel in Research activity so that they can function as an integral 
part of the field, and it suggests the wisdom of a strategy of collabor- 
ative/ interactive Research planning by Agency and field together. Under 
such conditions, the Agency vs. Field issue melts away. 

Development 

Development calls for a supportive style of relationship between 
Agency and field, with Agency personnel facilitating quality activity 
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where it exists and building or renting additional Development capacity . 
where it doe's not exist at all or exists at too limited, a level. 

" l>is8emination 

Dissemination is the except|pu to the general theme of needed 
close relationships between Agency and field. Given the large and 
fragmented nature of the field as. defined by this function, there is 
no way the Agency can relate directly to the field. Instead, the 
Agency must relate to the field through directing or facilitating the 
work of intermediaries. 

Evaluation Research 

The political necessity that Evaluation Research be done 
well and the Research aspect of Evaluation Research both indicate 
that NIE personnel must maintain a close working relationship witU 
the strong talent in the field. 
2. Multi-Purpose and Portfolio Emphases 

In all of the functions, we have emphasized that each individual 
procurement should be examined not only in terms of its manifest pur- 
pdfees but also in terms of additional ways the project could or would 
be likely to impact-various parts" of the R/D&I system. A premium should 
be placed o^ those projects which not only have important manifest pur- 
poses but which can also provide multiple additional impacts on the system. 

Similarly, the cost /effectiveness of a given project should be 
evaluated in terms of its overall impact upon and within the total 
portfolios 6f projects and programs at NIE, ^ross funding agencies, and 
within particular performer organizations. 

Research 

The key to effective Research procurement Is the development 
of a bet-placing portfolio. The substantive Resec 'ch outputs 
being procured must be recognized as having potential long term 
importance, with a very significant aspect of any procurement 
for NIE being its system-building potential. Thus, a Research 
portfolio must be understood as an area portfolio (a portfolio of 
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Research areas in Basic Research/ problem ateas in Froblem^Focused 
Research) . . \ 

, Development * 

Development portfolios must balance high-risk and low^risk 
projects, short-term and long-term activities, etc* Since the 
substantive outputs of the Development funct^ion take on some more 
immediate substantive importance in their own right, this balancing 
of multi-purposes takes, on particular significance. Wherever pos- 
sible, potentially synergistic projects should be funded at the 
same time (either by a single agency or in coordination with other 
funding sources). Additionally, multi-purposes should be served 
explicitly wherever possible and balanced adequately in an agency's 
overall Development portfolio. Finally, it is vital for a funding 
agency to be sensitive to the manner in which Its funding decisions 
shape the character and capabilities of the Development organizations 
with which they work. 

Dissemination 

In all other functions, we have suggested that portfolio/multi- 
purpose effects be considered along with the manifest purposes of 
a particular procurement. Because of the system-building nature - 
of Dissemination, portfolio/multipurpose effects must be considered 
as an inherent part of the manifest purposes of each procurement. 

Evaluation Research 

In Evaluation Research procurements, consideration must be 
given to portfolios which are likely to have long-term system build- 
ing effects* One key question, for instance, would be whether the 
bulk of Evaluation Research funding is flowing to those institutions 
most likely to produce high quality work ^md develop long-term 
capabilities, or whether instead it is flowing to organizations 
less likely to meet these needs — and what this suggests about 
any weaknesses of Evaluation Research procurements as currently 
structured* 

Additionally, consideration must be given to balancing the 
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varied needs of project managers, funders and Evaluation Researchers, 

Finally we roust note that Evaluation Research is inherently 
multi-purposeful. That is, it has^the three functions of: (a) 
meeting information needs of project managers; (b) meeting in- 
formation needs of funders; and (c) adding to the overall know- 
ledge base of the education sector. 

3, Staging 

Procurements might be designed in such a way that the carrying 
out of an overall project is broken down into stages (within and be- 
tween functions), providing the Agency with the option to award dif- 
ferent phases of the overall contract (e.g., prototype design/Develop- 
ment/ field testing/Dissemination) to different contractors with dif- 
ferent strengths, or to at least provide the possibility for field 
as well as Agency input and review at each step along the way. 

Research 

Research ; In Basic Research, staging strategies are likely 
to be irrelevant. In Problem-Focused Research staging may be slightly 
more feasible, e.g., by separating into two stages the Need/iden- 
tification/project selection process from the actual conduct of the 
Research. 

Development 

The issue of quality control provides the basis for staging 
in the Development function. On the one hand, quality control is 
vital to ensure that Users will be able to obtain "good" products and 
thus will develop trust in the R/D6el .system. On the other hand, we 
have noted the field self-quality control is weak. Thus, NIE must 
provide for quality control through the creation of mechanisms for 
monitoring of the various stages of the Development process. 

Dissemination 

The system linkage nature of Dissemination and the multlple- 
compjnent nature of effective Dissemination strategies lend themselves 
to a staging process. 

More importantly for NIE, building the Dissemination system 

^ tj ,j 
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in stages may be both wise and necessary because of the combination of: 
(a) the very large size and the variety of the User target group; (b) 
the cost of building a Dissemination system; (3) the weakness of our 
knowledge about Dissemination as a total process; (4) "eed to build 
"fail-safe" into the Dissemination system. 

Evaluation Research 

As noted earlier, the relative variability of the Evaluation 
Research community — make staging necessary. The political context 
makes staging seem a particularly advantageous strategy. Each stage of 
the Evaluation Research process requires different skills. The Research 
design stage is most critical of all and, requires bringing in the best 
design talent available to design meithodologically sound studies able 
to withstand the inevitable attacks on unpalatable findings — often 
attacks that are political in nature but are couched in methodological 
terms. The design stage also seems to be the critical point in the pro- 

r 

cess for identifying, orchestrating ,and taking l»to account the diversity 
of value-laden viewpoints that are likely to perceive a stake in the 
definition of "suitable" Research questions and determination of "ap- 
propriate" methodologies — to make certain that what any stakeholder 
might perceive as unpalatable findings is not predetermined by prlo^ 
choices of focus and methodology. 

Whereas in the design stage, the best design talents would seem 
to be required, the picture is rather different in the data collection 
phase, where substantial numbers of competent organizations are available 
and competitive procurement is likely to be appropriate. In the data 
analysis stage (and perhaps too in a subsequent data reanalysis stage), 
the best minds would again seem to be called for here the best analy- 
tical minds. 

Staging in this manner is likely at this time to produce work of 
far better quality than awarding a single contract for conduct of the 
whole process. Further, there is likely to be a substantial by-product 
in the form of strengthening comraunication mechanisms in the field (e.g.: 
among the best' analytical talent) and enhancing the cumulative development 
of the knowledge/ technology base of the field. 
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4« Process vs. Product Management: Approach 

Given the uncertaia charactp.^ of the field's knowledge/ tech- 
nology base and the professional, tr-vinitig and socialization of the 
available personnel, a tight mode of niAnagement of R/D&I activities 
is often not feasible, A more reasonable, alternative than tight 
monitoring of the R/D6el output or product is monitoring of the R/D&I 
process . This becomes an especially effective mode of management 
if NIE staff function as an integral part of the field while also 
stimulating the field's development of self-controlling mechanisms 
that make an Agency role in quality control less and less, relevant 
over time as the system matures. 

Research «} 

« 

Given the uncertainty inherent in the Research process, 
there is simply no way that the Agency can control the Research out- 
put. Rather,, control over the Research process is more appropriate 
and more feasible — and is most likely t'o be carried out well if 
Agency personnel are involved in Research activities, function as an 
integral part . ' the field, and have close relationships with the 
field. In Problem-Focused Research in particular, where the scale 
of the Research is likely to entail a considerable Agency investment, 
the need for control is high — especially since the capability of 
the field is not well developed. However, since this is Research it 
cannot be controlled bureaucratically. The process mode of management 
seems essential, but this does require internal Research capabilities 
within NIE and a pattern of relationships with the field that do not 
yet appear to be the general rule within the Institute. 

Dev6t3!opment 
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In relation to the Development function, the process 
mode ^fm^iiagement involves controlling who does Development work 
(through project selection) and to some lesser extent how it is done 
through the Agency Involvement in some aspects of process design. 

Dissemination 

In Dissemination, the process mode involves control by 



229 



ateeiring, guiding and overseeing the Dissemination process. 

Evaluation Research , 

The process mode of management within the Evaluation 
Research £iiact5.on involves exercising control at key points in ,the pro- 
cess. ?or example: 

1,. Intdrveaing during the design stage to insure that differing 
value-^X^dea perspectives are taken into account in the initial 
definitioa of the questions to be investigated and the approaches 
to be used; that evaluation questions are researchable; and 
that Research designs are methodologically sound and likely 
to lead to clear^ answers to the questions under investigation; 
2. controlling the types of Evaluation Researchers selected 

to carry out particular Evaluation projects with the more 
innovative (as opposed to the more conventional) programs 
calling for the kinds oi^ Evaluation Researchers who are ere- 
ative and who also have ah; in-depth understanding of the 
particular kind of program '^eing evaluated. 

5. Facilitating. Buying, or Rentjlne ' 



Where needed capabilities exist to f^some degree within the educa- 
tional R/D6cI system, a facilitating/ collaborative strategy is possible 
for NIE. However y where specific needed cap4billtles do not exist (or 
are too few), NIE is forced to be more directive. In that case NIE*s 
options would seem to be to "buy or rent*' the needed capabilities. The 
''buying** strategy would Involve a direct attempt to create Institutions 
with the needed capabilities that would become permanent and largely 
committed parts of the educational R/D&I system* A "renting" strategy 
would entail the temporary purchase of services fpom org4nlzations that 
already posess the needed skills^ but are external to the educational 
R/D6el system in that tl^eir involvement tends to be limited and/ or of 
short duration. In this case, a "gap" in the systeip is "filled", but 
only temporarily, on a project-by-project basis. The advantages of the 
"renting" option are that the needed capabilities are available immediately, 
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NIE can be rather directive about what is to be done, and the funding 
need be only short, specified periods. The disadvantage is that this 
option does not increase the overall long-term capabilities of the 
system. , , 

. Renting may be a preferred option in the case-^<>f highly spe- 
cialized or seldom used capabilities that need not be developed internal 
to the system. Additionally, renting may be a useful interim strategy 
o- so long as its use does not retard system development because the 
outputs of the emerging internal capability are not as polished and 
professional as those likely to be produced by external organizations 
with well developed capabilities that can be rented. 

Research 

Research : In Research "buying" strategies tend to be 
most appropriate at this tim^ since the primary consideration in 
Research procurements, we ha^e suggested, is system building. 
While renting is perhaps not -a very apt description in this instance, 
cases of attracting Researchei^s from other fields inuo educationally 
relevant areas for relatively phort periods would be analagous to 
renting. This has been going on in education but should not be a major 
strategy. 

I 

Development 

In Development, i^enting or buying are Agency options, 
especially with regard to the generalizing, packaging, production, 
and dissemination of exemplary programs developed In practice-based 
settings. The renting option permits rapid packaging and dissemination. 
The buying option builds long-term Internal system capabilities. 
Mixed renting/buying strategies are al^o aprrr.prlate. 

Dissemination 

In relation to Dissemination the need to build the system 
and the likely high costs of renting suggest the advisability of buying 
strategies. However, the easy repllcablllty of some Dissemination 
modes permits and even favors renting options under certain conditions. 
Mixed renting/buying strategies may be particularly appropriate at 
this point in the system's development. 
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Evifluatioa Research 

Here, ton, there are options. However the strong systen faci^a 
tatlng , and building needs suggest the advl9a})iIity -o£ emphasizing buying 
strategies as the preferred altarnat*ye at t^is tibe, even though opportun- 
Itles of. making use of highly qualified .^ad specialized organizations need 
not be abandonned as part of staging strategier . 

6. MfKed Strategies . ' • ' ' 

^ Wher^ the' State o£ knowledge about a fuactloa Is low. It may be 
most appropriate for an agency to pursue mixed strategies that provide 
alternative possibilities and a fair amount of redundancy that is. orches- 
trated subtjl^ rather than.being "over-coordinatexi^^. or "ovcr-mauaged". 
Often advisable is a* policy tl:^at permits a substantia?, degree of natural 
variation nlong with built-in mechanisms for monitoring natural field 
experimei;it9 and using documentation-and-analysis Research to develop a 
stronger ^knowledge/ technology base for the field. 

The need for 'mixed strategies gets stronger, the clnscJ: one 
gets to th^ User end of the R/D&I continuum, with nixed strategies being 
essentially, mandatory in relation to the pissetnination function. 

Researc^^ ' 

The use of mixed strategies is likely to be minimal in 
Basic Research, where "placing bets" on specific Research areas is 
likely to be the primary strategy used. There are ciore options in 
relation to Problem-Focused Research; e.g. : using small and large 
organizations; strong NIE personnel and advisory panels from the field. 

Development 

Mixed strategies are highly appropriate to the Development 
function; e.g.: mixing use of staging options;^ single vs, mvfltiple 
institutions; specialized Development organizations vs. practice-based 
Development; practice-based vs. technological-opportunity-type Need 
Identification; production by Developers vs. Users; etc.. 

■ Dissemination 

In Dissemination, the use of mixe^ strategies is mandatory 
given the variability among products, User needs, User capabilities, 
etc..* Mixed strategies are particularly appropriate here since there is 
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"best'^e 



clearly ao one "best" mechanism and many mechanisms exist or qan be 
created. Use of mixed strategies is the essence of designing fail- . 
safe .systems,. 

Evaluation Research 

The use of mixed strategies here is limited somewhat to 
staging^ and to selection of different, types of Evaluation Researchers 
(e.g.^ for evaluation of taore innovative vs. more conventional kinds 
of programs). The primary strategies must be:* (a) resolving politi- 
cal issues in the design stage; and (b) providing a balance among 
differing informati_on need^* 

IV. Ar COMPARATIVE ANALYSIS ACROSS VuNCTIONS ; SIGNIFICANT DIFFERENCES 



1. Time Frame 

Research 



The timeframes for Basic Research projects tend to be 
rather long, while those for Problem-Focused Researcl^ may be' some- 
wh''t more moderate. Further, sit. Research involves a high level of 
uhccrtainty, the Research time frr ... are generally not predictable. 
System building for both variants of the Research mode tends to take 
conciiderable time — both to' identify and train creative Research 
.talent within existing centers of excellence and to build Research 
teams with the needed longevity for producJ:ive relationships. The 
,^ate of expansion for both variants of Research is limited' by the 
number of such existing centers and their capacity to train additional 
personnel and absorb and use additional funds proauctively. 

Development 

The time frame for Development projects tends to be short 
to moderate in length and relatively more predictable. --a^ 

Dissemination 

Dissemination involves a very different concept of time* 
It requires consideration of the interrelationship between two different 
time lines — the rapid rate at which Dissemination capacity can be 
expanded and the much slower rate of User absorption and utilization 
of what Is dissemin^lt^d. To avoid unrealistic expectations for system 
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impact, the rate- of Dissemination capacity building must be regulated to 
keep it reasonably congruent* with expected User absorption rated. An- 
other option of course is to ^cilitate the work of organizations that 
provide the.kind^s of. User system technical assistance and support l^at . * 
might increase somewhat th6 rate of User absorption and utilization. 

Evaluation Research 

^ This Euoction tends to have extremely tight, highly spe- 
cified time lines tied^^to the informatioil needs of decision makers and 
the time frames of their decision processes. Since evaluation findings 
are gathered for immediate usage as input to these decision processes, 
the tine lines are major constraints on the Evaluation Research process 
and cannot be shifted at the initiative of the Researcher. They may . 
*dt times be even shortened at the initiative of the client, as in the 
case when information is needed more quickly than the time frame in- 
itially specified in the Evaluation Research procurement. Consequently, 
the need for interim reports becomes a matter of some importance as 
d6es the design of Research procedures that permit staged gathering of 
information aiid formulation of findings. 

There is often a tension in Evaluation Research between the ^ 
immediacy of information needs for decision inputs and the long-term 
nature of what is being evaluated — more often than not the slow change 
process experienced by individuals, social groups, and communities* The 
more socialized the Evaluation Researcher has been in other modes of 
Research (especially Basic Research but Problem-Focused Research as well), 
the more tension is likely to be produced by the time frames demanded 
by the Evaluation Research context. 

2. Excellence 

Research 

In Research, Excellence is the key criterion of judging 
projects and institutions. In Basic Research, Excellence is defined 
in terms of creative^ productive, rigorous activity at the outer limits 
of the state of the ^art. In Problem-Focused Research, the same 'definition 
applies, tempered to some extent by standards of practicality. 
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Development 

In Development, Excellence is defined in terras of what 

is feasible, practical and usable, not by the outer limits of the 

/ 

state of the art. j 

Dissemination 

In Diasemination, Excellence is defined in terms of 
effectiveness and professionalism. 

Evaluation Research 

In Evaluation Research, a substantial amount of tension 
surrounds- the Excellence issue. On the one hand, it has to be done 
very wall (although not in terms of the Evaluation Researcher seeking 
to break the outer limits of the state of the art). However, thef 
iniaediacy of information needs poses serious constraints on what the 
Evaluation Researcher can do. Excellent work that is concluded too 
lata to affect decision processes is not useful. Consequently, the 
Evaluation Researcher must produce the best work possible within 
specified time constraints. Interim reports in particular may need 
to be judged with a different standard of excellence from. the stand- 
ard used to judge a final report. ^ 

3. Key Criteria for Project Selection 

Basic Research 

For Basic Research in education at this time, the project 

selection process calls for: 

1. placing bets on Research areas that are likely to 
bear fruit; accepting the fact that there is likely 

to be no one "right" project; and becoming comfortable 
with a degree of opportunism in the selection of 
.Research areas (particularly being opportunistic 
about th«i Research strengths of the NIE Research staff) ; 

2. searching for centers of excellence possessing the 
needed critical masses of Research talent; and 

3. viewing system building considerations as paramount 
in project selection. 
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Problem^Focused Research 

In Problem^Focused R» earch, project selection criteria 

must include: 

1* insuring the existence of an adequate problem focus; 

2. insuring the interdisciplinary nature of the persp/c-\ 
tives brought to bear^ as needed; \ 

3. insuring the longevity of team commitment to the problei|i 
^ . area» 

Longevity is also important in Basic Research, but tends 
to be less of a problem because of the nature of many of the better 
Basic Research personnel C4.id their personal commitments to areas of 
lies earch* 

Development 

In selecting Development projects, the Agency must start 
from a posture of accepting the potential equivalence of an array of 
alternative projects. Thus, selection criteria must focus on: 

1. whether the projects fall in the priority areas 

2. whether or not the contractor has the needed capabilities 

and 

3* likely system impacts in terms of system-building, multi- 
purpose^ and portfolio considerations » 

Digsemination 

The key project selection criterion in Dissemination are: 

1. cost/effectiveness in relation to multi-purposes and 
portfolio effects; 

2. providing for fail-safe* 

Evaluation Research 

In Evaluation Research, project selection criteria include: 

1. researchabillty; 

2. likelihood of providing information in a form that is 
reasonably immune to methodological attack to serve 
political purposes; and 

3* system-facilitation considerations. 
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V. A CCKIPARATIVE ANALYSIS ACROSS FUNCTIONS; NON "PROCUR EMENT STRATEGIES 

AND NIE PERSONNEL ^ 
1. Non -Procurement Strategies 

^ 

We have noted that NIE should consider £hree general types of 
strategies: Procurements, Non-Procurement Behaviors, and Internal NIE 
actions. The common themes we have been considering are primarily 
procurement-related but can also involve non-procurement behaviors. 
For Instance, we have mentioned at several points the possibilities 
for NIE to orchestrate and coordinate funding across the various fund- 
ing agencies — e.g.: federal agencies; fuxfding from other levels of 
government; and other potential private sector sources of funding. 
Multi-purposes and portfolio emphases are as appropriate to the fund- 
ing coordination process as to the examination of individual procurB- 
meats or Agency program agendas. Coordination of funding may be used 
to illustrate non-procurement strategies across functions. 

Research 

In Research, cross-agency coordination is vital because 
much educational Research is likely to be interdisciplinary in nature. 
Further, while the proportion of total Research funding provided by 
NIE Is small, NIE is the lead agency. 

Development 

In Development, coordination of funding across agencies 
is likely to be helpful but is not so essential. It may be adequate 
simp^ to know what kinds of Development projects other agencies are 
funding and therefore where the potential for synergy may exist. 

Dissemination 

In relation to Dissemination, coordination across agencies 
may be helpful. However, OE rather than NIE is the lead agency for 
activities that relate to the operating system. Consequently, NIE 
may have to work very closely with OE in these coordination activities. 

Evaluation Research 

Given the extensive amount of Evaluation Research activity 
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that Is supported by federal agencies and other levels of government, 
coordination of such funding would seem to be helpful both for orches«» 
tration purposes. ;and also for developing a cumulative knowledge/ tech- 
nology base for the Evaluation Research function. 

2. NIE Personnel 

We have noted that to orchestrjte, monitor, and relate to the. 
field in each function, NIE needs ^toi have personnel with specific types 
of skills and backgrounds. 

Rese arch 

If NIE is to relate to the field in the manner we have 
suggested, it will be.essential for NIE to have on its staff Researchers . 
who are from the field and who continue to be involved actively in the 
Research process. This would seem to be an essential rkquirement for 
effective NIE process monitoring and orchestration of the field, and 
even mor^ critical if Research planning is to be i^arried out in an 
interactive/collaborative n^ode. We have suggested that the kinds of 
Researchers needed here are not the "stars** of the field who are likely 
to distort the field to their own image of where the field should be. 
Rather Researchers are needed who can facilitate and work with the field, 
who are sensitive to d<^velopments going on throughout the field. These 
requirements would seem to contribute significantly to the case for NIE 
conducting a limited in-house Research program not in competition with 
but as a contributor to the field. 

Development 

The kinds of NIE personnel needed to manage the Institute's 
Development programs would seem to be education professionals sensitive 
to the needs and constraints of the edut ation context and who also possess 
the kinds of skills needed for effective orchestration and facilitation. 

Dlsgeminatlon 

If the Dissemination function is to be carried out in a 
highly professional manner, it would seem advisable to have at least some 
Dissemination professionals on the staff of the Institute — talent that 
can likely be ''rented*' temporarily or acquired on a more permanent "buying" 
basis.. 
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Evaluation Ilesearch 

The Internal Agency skills needed herS would seem'' to be 
political as well as methodological insight and savvy regarding such < 
matters as different vjrays of defining problems; the different kinds 
of Rasearch methodologies that are relevant to different kinds of 
projects; the vai^ying needs of program managers , funders, and Evalu- 
ation Researchers; and the relevant potential political implications. 
Most likely this mic of skills is to be found only in experienced, 
first-rate Evaluation Research talent. ^ — 

In concluding this section we feel that it is appropriate 
to echo a theme first raised in the Preface. NIE priorities and re- 
sponsibilities can not be only a reflection of the Agency's budget 
profile. As a small agency, in a large field its activities as co- ,^ 
ordinator, orchestrator and facilitator, involving non-procurement 
based efforts dre likely to have very significant implications for 
the short and long term health and functioning of the educational 
R/D&I system. > 

3. Structures 

A cross-sectoral comparative analysis reveals some differences 
across functions in the forms of NIE organizational structures that would 
be most relevant to each function. 

Basic Research 

The uncertainty involved in working at the outer limits of the 
state of the art is the key to determining relevant NIE structures in re- 
lation to Basic Research. NIE cannot orcHestrate, guide and monitor this 
field from centralized structures and detailed plans. Rather, NIE needs 
to have personnel who know the field: what\ is being done, where and by 
whom, in order to determine which areas of Aoncem to concentrate upon and 
which personnel and institutions t6 support in a long-term system-building 
mode. These conditions suggest decentralized, emergent-organic structures, 
with generally open boundaries between NIE and the field such that NIE 
personnel are actually a part of the field. This would seem to call for a 
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relatively loose distinction between NIE/field organizational membership for 
NIE personnel — for example, as might be aqhieved through personnel inter- 
changes and leaves to and from universities (as is now occasionally done) . 

Problem-Focused Research 

In comparison to Basic Research, there is relatively less un- 
certainty in Problem-Focused Research, though this is still a significant 
factor. The boundaries between NIE and the field are clearer, but there 
remains the need for NIE personnel fco be directly involved in Problem- 
Focused Research. Since both NIE and field personnel would be involved 
in Problem-Focused Researph (though probably separately), the key element 
of structure becomes some form of linking mechanism between NIE and the 
field. A matrix structure might be relevant in terms of the Interdisci- 
plinary nature of educational^ Research an<^ the various problem areas of 
Problem-Focused Research — and to facilitate selection of institutions 
to support in a system-building mode. 

Development 

Development is not an tn-house ^iJiE. function. NIE*s role is 
to provide direction through funding and laonitoring. Since Development is 
a product-oriented function, NIE might consider structures which are organ- 
ized according to product typologies. 

Dissemination 

Because of the size and variety that characterizes education 
and because we generally cannot say there is "one best way" for all situa- 
tions (or even for a particular situation) , diversity and even redundancy 
characterize DissemiPatlon. Thus, there will be structural diffusion — 
i.e., several different structures may emerge in the field and be ^supported. 
NIE structures must be supportive of, and congruent with, field structures. 
One relevant NIE structure could be a problem/geographical matrix 'organiza- 
tion. 

t * * 

Evaluation Research 

The key needs for Evaluation Research are (a) orchestration of 
complex set of needs, relevant participants and perspectives; and (b) insur- 
ing that Evaluation Research is done well. This situation presents a key 
structural dileonna. 
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On the one hand, the politically sensitive nature of Evaluation 
Research requires that it be done well and that potential political issues 
be dealt with in the Evaluation Research process. Further, the various and 
possibly conflictin'g needs and perspective^ of the participants require 
orchestration. Finally, especially with regard tb summative Evaluation 
Research, it is the policymaker/funder who must and will determine problem 
definitions and Evaluation Research objectives. These conditions would 
seem to call for close control at a hi§h level within NXE. This would also 
seem to call for top level NI? personnel to have both substantive and 
methodological skills and political sawy. 

On the other hand, the Evaluation Researchers need to have a 
good deal of freedom in matters of Resiarch methodology. Further, there 
need to be ways for the Evaluation Researchers , to provide guidance in mat- 
ters of problem definition and Evaluation Research objectives. Additionally, 
program administrators need to be able to obtain the information they need. 
These conditions would seem to call for significantly less control by NXE. 

Perhaps it is best to say that N.IE must insure that basic needs , 
are met by the Evaluation Researcher (e.g. : that some level of basic 
Evaluation Research standards are met; that certain NIE-specif led objectives 
are accomplished; and that the 'political issues are dealt with construc- 
tively — but that beyond these basic requirements, there should be a good 
deal of freedom and flexibility within the Evaluation Research function. 

VI. CONCLUPING REMARKS; COST CONSIDERATIONS IN RELATIO N TO POTENTIAL 
LOyiG-TERM INVESTMENT PAYOFF 

J 

We would be remiss in concluding this analysis without mentioning 
the costs likely to be incurred by the kinds of strategies we have proposed. 
In comparison to the procedures currently used, it ii> likely to be con- 
siderably more internally costly for Researcu. prograirs to be developed 
through long-term, intensive, collaborative/ interactive relationships be- 
tween Agency staff members and the field. To take another example, stag- 
ing of procurements (especially when this involves complex orchestration 
of diverse viewpoints in some stages) is likely to appear to be a more 
costly alternative thin awarding a contract for the whole project to a 
single institution. 
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We acknowledge this cost factor. But Wjfe alsq point out that the 
considerable amount of NIE staff time currently invested. In the research- 
ing and writing of RFPs was one of the concerns that prompted the.Council 
and the Institute to request 'this policy analysis. The added cost may 
also be more apparent than real when this is offset by fiotential gains in 
productivity. 

We have ho way of putting, a price tag on the kinds of options we 
have proposed. But if NI9 accepts the mission and the ro.le we have sug- 
gestied, the costs will have to be absorbed and referenced in terms of both 
increased effectiveness and efficiency and ^he potential long-term iArest- 
ment payoff in building future aystejn capacity/ The point need not be 
labored further. • ^ - 

The case for NIE to assyme system-oriented res;)onsibilities Is, 
we believe, a fitroug one, entirely consistent with its legislative mandate 
and its position as lead agency for Research ai^d Development in education. 
We suggest that the rather fundamental and broad-rang i.ng questions raised 
by the Council and the Institute, once subjected to analysis from an R/D&I 
syst^e perspective, demand at the very .least' thoughtful consideration of 
the case for a rather substantial restructuring of the Agency's relation- 
ships with the field. 
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SCENARIO ANALYSES 



NIE-DIRECTED COMPETITIVE PROCUREMENTS! A SCHOOL DISTRICT SURVEY 

1. Description of the Procurement 

2. Impacts 

3. Evaluation of the' Procurement and its Impacts, Actual and Potential 

4. Possible Alternatives , , * ^ 4*4 

5. Summary Descripticn of Alternative Procurement and Related Activities 

6. Likely Impacts 




.1. Description of the Procurement 

2. Impacts ^ , j « ^ ..j - 

3. Evaluation of the Procurement and its Impacts, Actual and Potentiax 

4. Possible Alternatives , * 4^4 

5. Summary Description of Alternative Procurement and Related Activities 

6. Likely Impacts 
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SCENARIO ANALYSES 

INTRODUCTION \ 

♦Two illustrative scenario analyses follow: 

1, an examination of the implications of alternatives to 
NIE-Dlrected Competitive Procurements, where less 
ccMapeti'-ive modes might seem more appropriate; and ' 

2, an examination of the unsolicited proposal as a tiro- 
curement mechanism and options that might be consid- 
ered in funding R/D&I activities initiated through 
the unsolicited route.. 

Consideration of these two issues in particular was requested by NIE 
staff. 

The specific scenarios we have developed both concern procure- 
ments of Research. In part, this is because both issues seemed to 
lend themselves to richest illustration in the Research context. But 
also, it seemed to U5 that Research thinking is central to NIE pro- 
curements as- they are made currently. If we could be persuasive in 
making a case for system-oriented thinking in relation to the Research 
function(where much of what we have been suggesting might seem particu- 

1 

larly alien) , then it might be even easier at a later date to make a 
case for this pattern of thinking in relation to other functions. 

Each scenario is presented in accord with the analytical model 
described earlier in our introductory chapter (see pp. 19-21 and es- 
pecially Figure 3): 

^'^ 1, a description of a procurement (in both cases these , 
are hypothetical procurements but based to some degree 

on typical ca^^es ) ; 

2, analysis of the likely impact of a procurement im- 
plemented that way, given the educational R/D&I context 
at this point in time; 

3, evaluation of the possible strengths and weaknesses of 
the procurement based <?n the above analysis; 

A. development of alternati/es to (or modifications of) 
the initial pattern of procurement that might better 
achieve Agency purposes; 
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5. description of the mpdlfled "r alternative procurement 
approach, its latent and manifest purposes, and Its 
implications for non-procurement activities and actions 
within NIEj and / 

6. analysis of the likely impacts of this alternative pat- 
tern of procurement. 

It should be noted that both scenario analyses presented here were 
hypothetical rather than actual cases. Clearly, the scenarios would be more 
useful if they were based on actual cases of NIE procurements — past, present 
or contemplated. We would have preferred to develop a series of scenarios 
based oij actual cases. However, this would require some discussions with NIE 
project officers and other NIE personnel, as well as analysis of relevant 
documents and perhaps too, discussions with others outside of NIE involved in 
the planning and/or implementation of the particular procurement. We would 
hope to conduct such empirically-based scenario analyses some time in the 
future if NIE judged this to be desirable. 
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I. KIE-DT.RECTED COMPETITIVE PROCUREMENTS; A SCHOOL DISTRICT SURVE Y 
1, Description of the Procurement* ^"' 



In 1975, NIE Issued an RFP for a survey of 2000 school Districts. The 
objective was to be able to identify those school Districts that may have had 
some involvement in lo ally generated innovation. The survey was to gather 
^^slc descriptive data on these "school Districts, who they were and what kind 
of things they had done, size, age, location, personnel, student population, 
SES characteristics, etc. — so as to develop a better understanding of the 
conditions which affected the innovation behavior of those school Districts 
that would be so identified in regard to both Internally and externally devel- 
oped innovations. One of the central questions of concern was whether these 
particular school Districts, having already demonstrated some capacity for 
self help might not represent a highly receptive and possibly qualified pop- 
ulation for R&D products that had been developed externally. If so then, 
poten'ially, these L -tricts might well merit becoming the target of more in- 
tensive dissemination and technical service efforts to supplement their own. 
In turn they might become ^he source of model programs, of exemplary practices. 

On the other hand, it was also possible that, given their own, self 
generated efforts they would be resistant to innovations deriving from ex- 
ternal sources (the Not-Invented-Here syndrome).' Further, it would be im- 
portant to know whether such conditions held generally (or not) across the 
total population of such school Districts or whether they might vary with 
respect to such factors as: 

size 

urban/rural 

miuority concentration 



*vmile the case is presented to read as though it describes an actual NIE p 
curement it is in fact hypothetical. Also the authors imply no position on 
substantive content of the procurement. 
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professionalism of staff 
District structure etc* 
Critical dependent variables to be observed as indicators of 
potential receptivity might be: 
^ scope of adoption behavior (rates, scale, etc*) 

maintenance a^f ter initial - trials 

observable levels of Instituuionalization (e.g., after 
periods beyond 3--4 years) 

The study was seen as, the first step in what might become a long 
term survey program aimed at monitoring these effects so as to permit 
a responsive strategy on the part of the various delivery and dif- 
fusion mechanisms that might be operating In relation to these Districts* 
Thus 9 this Initial survey was viewed as a critical first step that 
would develop ^he sampling frames that would permit improved later 
studies and also both natural and planned field experiments (e.g., 
to examine alternative dissemination and/or technical assistance 
strategies)^ as well as to pefmit experimentation \d.th varying modes 
of Development (e#s., specialized Development Organizations vs. 
Fractice*6ased) « 

The core of the issue was how local innovation activity inter*' 
acted with external based R&D. A particular sub-concern involved the ^ 
special case of **Baslc Skills". How might the conditions implied above 
affect the likely response to product^ being develpiped in this area 
in the R&D system and, relatedly, what could be dotle to make such 
products or programs more adoptable and useable by this type of 
potentially very important target (in terns of the planning of a 
major development and delivery program)? 

The survey was to be conducted in two phases 

Phase 1 - Survey of the 2000 Di'stricts selected as a carefully 
statified sample of the total population. The objective of 
this phase was to identify which of the Districts had exhibited 
what kind of internal innovation behaviors, and th^ir ]>fehavior 
with respect to externally based R&D products. 
Phase 2 - Was to be a more intensive survey of the sub-group 

identified in the phase 1 survey as having had a history of some 

) 
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s.ignificant internal innovation activity. The objective of this 
phase was to increase our knowledge of the nature behavior and 
attitudes of such Districts* The size of the second sample would 
be determined based on the phase 1 findings -'but with the hope 
\of having a population of some hundreds. v 
Based on the two phases it was hoped that the characteristics of 
school Districts likely to have jbeen involved in different kinds of 
Innovation, with varying intensities, -could be identified, as well as 
specifically identifying a sample known to have been engaged in such 
'practices* Further , some specific Indications might then be obtained 
as to the responsiveness of this sub-group (and hopefully therefore of 
the larger group of school Districts that had demonstrated self«-help 
capacities) 7< ' . 

The primary purpose of this procurement was substantive ±.e.^ 
to gather the desired data. But there were other purposes involve^ as 
well; system-oriented purposes i±xi terms of developing baseline data 
for system monitoring and developing a listing of innovative school 
Districts) and environmental purposes (e»g.» demonstrating that 
the program was res|Sonslve to the manyjand diverse concerns 
that had been expressed by associations of Chief State School Officers 
and other LEA personnel who would make up a substantial proportion of 
the overall survey population) • 

A point to be noted about this procurement is the extent to which 
it partakes of characteristics of both Problem-Focused and Evaluation 
Research (or Policy Research) » It was Intended to Increase our under- 
standing of a potentially important group of Users/Innovators in the 
educational R/D&I system (about whom we know relatively little at this 
time), to permit NIE to carry out its system management role more ef- 
fectively. And too, it was oriented toward meeting information needs of 
policymakers suggesting an opportunity area in the system in need of NIE 
policy initiatives, and providing the beginnings of a time-series data 
base against which to evaluate the impact of the Institute's policies 
and programs. Consequently, such a procurement would seem to call for 
a balancing of the requirements of both Problem-Focused and Evaluatioii 
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Research a tension not easily mediated between a) the creativity 
needed to develop a useful picture of a little understood system and 
b) Agency specification of the key Information needs to be met* 

In the case o£ this particular procurement* the Agency Issued a 
highly specified RTF specifying the variables to be studied » the 
questions to be answered * etc* and requiring frequent and extensive * 
highly specified Interim reporting In accord with a specified schedule* 
A considerable amount o£ NI£ staff time was Invested In the preparation 
o£ the RFP (as well as In the .prior work to develop the stratified 
sample of the 2*000 school Districts to be surveyed In phase 1). 
There was some Interaction with SEA/L£A sector representatives (whose 
cooperation was essential to achieving a high survey response rate)* 
but little If any Interaction between NIE and other highly knowledgeable 
participants In the educational R&D system — * this despite the fact 
that NIE. had some contracts with a number of organizations studying 
aspects of the educatlonal^R&D system and specifically conducting 

surveys of school Districts ~ perhaps out of concern that such interaction 

with these contractors with whom the Institute already had a relationship. 

would be construed as giving them unfair advantage, — a violation of the 

"fairness" principle so strongly clung to in government procurements, where 

all potential contractors in a system are expected to be given absolutely 

equal treatment in their relationships to the Agency. We shall return to 

this to this point shortly. 

The RFP appeared to be a call for a s ingle » well qualified 

survey organization to carry out a predesigned survey and provide 

prespeclfled data analyses* One month was allowed for the response* 

2* Impacts 

a) The Agency received a substantial number of proposals in 
response to the RFP, mostly from single contractors (rather than 
collaborative contractor/ subcontractor arrangements) , mostly proposing 
to cariry out the work as specified rather than suggesting alternative 
types of surveys, mostly from large-scale private sector survey research 
* corporations — in short the competitive mode was reasonably successful 
In attracting a good number of the types of organizations considered 
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most appropriate to carry out this procurement. 

b) Costs: NIE invested considerable staff time in the stages pre- 
liminary to issuing the RFP and in the evaluation of proposals to select 
the survey contractor. Perhaps 20-30 organizations expended resources 
(equal perhaps to 1 to 2 man-months each) in responding to the RF? and 
in subsequent contract-related interaction with NIK. 

c) The survey contract was awarded to a single highly qualified 
organization with strong survey capabilities and substantive knowledge 
of issues relevant to understanding the innovative behavior of schools 
and school Districts. Agency purposes, then, would seem to have been 
achieved well. 

3. Evaluation of the Procurement and its Impacts, Actual and Potential 

In evaluating the procurement, Agency personnel must weigh actual 
(or likely) impacts of a procurement as initially formulated with possible 
desired impacts. The question to consider here is; How might such a 
procurement be designed to better achieve Agency purposes, or to achieve 
a broader range of Agency purposes? 

At the very least this question requires insuring some clarity on 
Agency purposes: What in fact are the Agency purposes that are viewed as 
relevant to this procurement? Might a somewhat different view of the 
Agency role nnd its requirements for procurement policies produca a 
different or an enlarged picture of the relevant purposes? 

If (a) NIE's role is conceived largely as being a funding channel, 
with Agency leverage on the system exercised primarily through choices 
as to which work it fund^, and if (b) the work to be procured is relatively 
straight-forward data-gathering in accord with the prespecified design — 
then the procurement mechanism (the BPF) would seem entirely appropriate. 
However, if NIE conceived its role primarily in terms of long-term 
system building, and if (b) the Agency started from the premise that 
the extent and processes of innovation in school systems are not well 
understood phenomena and that developing a data base for understanding 
such processes required the input of creative thinking from the R&D 
field and not simply more mechanistic kinds of conventional data-gathering 
■ (which is mora appropriate to well understood phenomena) — then other 
Agency procurement strategies become reasonable alternatives to consider. 
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If the survey is viewed as primarily data-gathering, i.e», to 
identify 

a) d specific group of **8elf help*^ innovator schools 

b) the structural and other characteristics of such schools 

c) limited and minimal data on prior adoption behavior in the 
respect to external innovations 

then competitive RFP>*type procurement may ndeed be the wisest course 
of action. Strong Research talent might be "turned off" by the RFP's 
specificity, but this might be quite appropriate since the Agency might 
view it as undesirable to divert strong Research organizations into con- 
ventional data-gathering of this type. Given the substantial number of 
competent survey research organizations attuned to RFP procurements who 
could effectively gather the desired data, Agency purposes would Indeed 
be well served by use of a procurement mechanism likely to attract these 
survey organizations and not the strong researchers whom the Agency 
would rather see doing other kinds of work for which they are more 
uniquely qualified. 

However, if the survey is thought of in terms of designing a data 
base and monitoring system that will be useful in the long run for de- 
veloping an und'^r standing of a weakly known and little understood 
group of school Districts, and developing the kind of understanding 
that will be useful for identifying policy-relevant leverage points 
as well as achieving a degree of system building, then it would seem 
that a case could be made for a different pattern of procurement that: 

a) could attract strong Research talent to the design stage of 
the survey effort; 

b) could have some impact on strengthening such strong Research 
instltutious and shaping their long-term agendas and port- 
folios; 

c) devel<^ing powerful communication linkages among the Research 
talent that might be considered an "invisible college" on 
school innovation processes and possible, synergy or collaboration 
aoong their separate efforts, 6tc»; and 

d) developing a close positive working relationship between NIE 
and this part of the field, with NIE's internal personnel 
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coming to be viewed as an integral and key part of the field, 
facilitating quality work in the field and development of the 
field. 

(If the procureaent is seen in somewhat creative terms, then NIE 
should be fully cogzizaat of an impact that seems entirely likely if the 
work is procured as it was in a tightly specified RFP. Such procurement 
forces creative Researchers with procurement savvy and understanding of 
the complexity of the task. they are undertaking into a form of game- 
playing with the Ageacy: contractors suggest that they will generally 
abide by the RFP's specif ications — for they will not otherwise win the 
contract — while in their own minds they are fairly certain that the com- 
plexities of the task make such scheduling, specificity of outputs, etc. 
infeaslble and likdj to be modified substantially in the course of conduct^ 
ing the survey). 

If NIE procurenears in general came to be viewed in multi-purpose 
terras, and in relaticrn to possible, supportive non-procurement activities 
and internal NIE actions, then the procurement might be carried out in 

conjunction with: 

a) NIE initiatives to coordinate its survey with similar work 
elsewhere (e.g., in OE) , to seek cross-agency collaboration, 
synergy, poolios of resources, exchanges of information, etc., 
and 

b) internal m actions to promote a systems perspective among 
personnel in program units and to develop a pattern of cross- 
program cocntuiication and synergy that might strengthen Institute 
functioning as well as total NIE impact on the R/D&I system 
toward which its somewhat discrete programs are directed. 

Thus for exampls, in this particular area a numbr.r of NIE groups 
would (or should) have substantial interest. These would include those 
concerned with Disserdnation and Feed Forward processes, with R&D utili- 
zation, with local problem solving, with the overall R&D system and with 
Basic Studies, even though only one of these groups, was actually involved 
in the procurement being discussed, and was in fact pursuing the project 
independently of the others. 



A* Possible- Alternatives 

A. Developlng^^i^ close working relationship with the strong talent in 
the field durl^ the survey design stage; 

Use of the RFP mechanism for procuring this work probably had the 
effect of cutting NIE off^trom the base of creative Research talent 
who. tend to be "turned off" by RFPs — who could not conceive of carry- 
ing out .Research thiey did not t^mselves design (or at least play a 
role In designing) or in working within tight specifications as to 
deadlines, outputs, etc. Many of chese Researchers may not even- be 
aware of the existence of this Research program (so that they gdeht 
think about participating) since they ai;e unlikely to read the Com- 
merce Business Daily or other such sources where competitions generally 
more attractive to entrepreneurial organlzi^tions are announced. 

2^ the procurement is thought of (Initially, or after the evaluation 
stage described above) as requiring input from creative Research 
talent to zero in onVhat kinds of things we need to study so as 
to achieve long-term polity-relevant goals (fo^r example that might 
Involve the exploitation of diffusion processes \based on seeding school 
districts having high implementation success probabilities, based on 
their demonstrated self help capacities)* ••- then\perhaps the survey 
design stage could be viewed as a wedge for achieving a wider range 
of Agency purposes. By bringing the strong talent Un the field to- 
gether in an ongoing relationship with the Agency, NIE might use the 
design of this survey as the basis for lon^-term program planning of 
the Research program to be served by (and to further the development 
of) the monitoring system. This kind of relationship with the field 
might affect not only the long-term NIE Research program agenda/port- 
folio, but also influeiiice the shaping of long-term agendas of these 
strong Research organizations. The greater the involvement of these 
organizations in the design of this long-term data system, the greater 
the likelihood that synergy will develop within and between their own 
organizations' Research agendas. Thus, this mode of relationship has. 
the potential, of being the wedge whereby NIE may shape the portfolios 




*We wish to remind the reader of the hypothetical character of both 
the process described and our neutral position as to the substantive 



content. 
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of the instltutioas that appear to have the strongest potential for 
producing important quality work in this area. The capability-building 
potential here may in fact have greater long-term consequences than the 
particular survey (or even the particular monitoring system) being de- 
signed*^ \ 

If a survey of this kind is to be an element in a continuing 
process, some kind of ongoing close relationship with the field of po- 
tential contractors might be essential. Otherwise, each contractor 
carrying out a piece of the long-term program is likely to pursue its 
own conception of what is involved here (which may at times be appropriate, 
but within bounds of some sort) , or NIE will be forced to be highly directive 
each and every time about the work to be carried out. 

B. Staging the procurement with competitive RFP-tvue proc urement used 
only in the data-gathering stage ; 

The procurement might be staged so that (1) R&D specialists 
work collaboratively with NIE during the design stage (either as "con- 
sultants" or as the result of modified competitions such as invitations 
to them to submit proposals of design ideas, etc.); (2) the larger 
scale phase 1 data-ga. nerlng stage is handled competitively through 
RFP-type procurement; and (3) the more intensive but smaller scale 
phase 2 survey and the dat;a analysis and subsequent reanalysis stages 
might again be handled in only modified competitive form (e.g., inviting 
specific innovation process specialists to submit proposals) or as 0|en 
competitions that include an element of actively encouraging certain 
organizations to submit proposals. With regard to the analysis stage, 
what we are suggesting may again come into conflict with the "fairness" 
principle. If some form of phase 2 and/or data analysis competition 
pits specialists involved in the design stage and/ or the phase 1 stage 
against the rest of the field, it is highly likely that someone will 
cry "foul" and argue that any of those involved in the initial efforts 
should be barred from the later analysis competition because their know- 
lecge gives them "unfair" advantage. The other possibility — closing 
this competition to only those involved in the design stage (or perhaps 
specifically inviting them to bid) might produce even louder cries of 
"foul" since clearly they are being treated differently from the rest of 
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. the universe of potential competitors. The problem 13 not easily 
resolved* 

C> Issuing .a competitive procuretnent^announcetnent that uses the NIB** 
developed survey design as Illustrative, and encourages creative 
design work and collaborative arrangement? between organizations 
strong in survey research and others strong in under s tend Ih;^ 
Innovation processes 

Another option rather different from ^at we have been suggesting 
would be to implement the procurement as a single procurement but 
change the specificity) time frame, etc* to make it possible for more 
creative design work to be carried out in the proposal development and 
to facilitate the development of collaborative arrangements among 
organisations with talents suited to different phases of the overall 
procurement. 

It should be noted that while collaborative contractor/sub- 
contractor arrangements were not prevented in the initial RFF, nor 
was a contractor prevented from suggesting an alternative design for 
the survey (in addition to bidding on the design specified), such 
possibilities were made somewhat difficult by the one month or less 
available to the contractor to draft the proposal* Collaborative 
arrangements take time to develop and complicate the problems of 
writing a coherent proposal* Furthermore, developing alternative con- 
ceptions for the survey different in conception, design, execution, 
etc* from what was suggested in the RFP — is likely to be difficult 
if not impossible within the RFP response time frame, especially given 
the varied other demands on the time of those drafting the responses* 
In all likelihood, considerably more time was Invested by NIE personnel 
in the development of the design specified in the RFP* Clearly, any 
organization responding to a competitive procurement is aware that 
Investment of time and money in responding to an RFP is a gamble with 
payoff likely only in a fraction of all cases. Therefore, the time an 
organization is willing to invest in responding to an RFP is likely to 
be limited — • not sufficient to conceptualize, elaborate, and present 
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design for an aUertiative approach significantly different from 
that sp .cified in the RFP* 

There are problems inherent in this option, then, and perhaps a 
staging here might resolve- this e.g., competitively procuring brief 
statements of creative design work proposed^ then selecting from these 
a small number of organizations to work with internal NIE staff to develop 
"the designs~with£n a~re"asonabre time frame. ' 
Coordination within NIE 

Given the fact that the survey data to be gathered have potential 
utility across NIE program boundaries, and that some of the data needs 
of other NIE programs might have been met by this survey, then planning 
of this survey might have been used as a wedge to promote cross-^divisional 
and cross -program communication within the Institute—to promote a more 
systems-oriented pattern of thinking across the Agency as a whole, 
especially to seek synergy and interaction effects across program lines. 
Clearly, the Institute would be well served by such a strategy. ^, 
4, Summary Description of Alternate Procurement and Related Acti vities 
A. Staging/Field Agency Relations 

The procurement could be usefully thought of as having the following 

possible elements: 

1) the initial conceptualization and general research design 

2) the first phase mass (2,000 school districts) survey 

3) data ajjalyais and the selection of the sample of "self-help" 
school district innovations . 

4) the second phase intensive survey 

5) the data analysis for the second phase and possible reanalysis 
of the first phase data 

6) design of alternative delivery strategies 

7) design of potential field experiments. 

While these elements would not all need to be so separated, it 
is readily evident that different Skills are required as between various 
combinations of elements. Also, there are likely to be critical questions 
that will need answering at' each stage before it will be obvious as to 
whether it is justifiable to proceed. Thus it will be only after element 
3) above (the phase 1 data analysis) that we are likely to know whether 
it is indeed feasible to identify self-help innovation districts through 
such a survey; whether there are any .reliable predictor characteristics 
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that can differentiate schools not only as to the general question 
but as to the types of innovation in which they engage; whether a 
sufficient number of such districts can be identified so as to make 
phase 2 viable; and most important, whether the phase 1 findings* 
give any indication of the "self-help" factor as likely to account 
for a variance ill tht reception to external R&D products. Similar 
milestone requirements Vi-)uld exist after elements 1) (initial design) 
and 5) (phase 2 analysis). 

These two aspects, thv^ variable skills required, and the 
critical milestone requirements lead us to consider staging as a 
potentially desirable strategy, as follows: 

Stage 1 , Initial conceptualization and research design could be 
carried out as part of a loosely defined grants compe- 
tition. The NIE staff would take responsibility for 

synthesizing the outcomes in collaboration with a panel 

of consultants drawn from the several Researchers (or groups) 

that were chosen from the competition. 

Stage 2 . An RTP, relatively tightly drawn and based on the initial ' 
synthesized conceptualization, would be used to select and 
fund a single contractor skilled for the phase 1 survey 
and preliminary data analysis. ^ 

Stage 3 . After completion of the phase 1 survey and analysis the 

NIE staff, in conjunction again with the panel from Stage 
1 above, would make the determination as to whether to 
proceed with the study. 

Stage 4 . If the decision was positive a second RFP would be issued 

for phase 2 and the continxiing analysis. This RFP would be 
far less tightly drawn— only mandating use of the phase 
1 results in the planning and design. These results would 
^ be a part of the RFP as would be the original synthesized 
conceptualization. The RFP should be open to everyone 
(possible) and collaborative relations between groups strong 
in conceptualization and in survey research eould be 
encouraged by indicating in the RFP the importance of 
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creative conceptualiaation of the issues and by permitting ^ 
the maximum possible period for response (and informally 
by NIE letting the field know that this is seen as a 
desirable aspect). 

Stage 5 . Using personnel identified in the previous stages as 

active consultants, NIE sould take responsiiiility f or ^he 
development of alternative delivery strategies. RFP6 
could later be Issued for the design and conduct of field 
experiments and evaluation research (formative and then 
susnatlve) when this becomes appropriate • 

B. Multl^'Purpose Strategy \ 
As we have already noted the opportunity for achieving several 

NIE purposes through this procurement exists. 

The manifest purpose of Identifying the "self-help** Districts 

and of learning something of their Innovation receptivity behavior 

has been described. But there are othe^ more latent: purposes that 

could be achieved. 

1. System Building 

At least tw opportunities for system building can be ideritified* 
First, the very targets of the survey, the self-help Districts, 
could be looked upon as potentially very important members of the 
educational R/D&I system (as sites for model programs, possibly 
receptive entry points for R&D products, models of local implementation 
capacity development, etc.). It might therefore be worth considering 
the development of a loose network of such Dls tricts--operating much 
as an invisible college might. Attempts to involve them in programs, 
even in later stages of the present study, could have positive consequeuces 
for the R/D&I system. 

Second, we could consider the consequences for the R/D6tl system 
of developi^iig the network of Rasearchers who might be involved in 
this study. The effect of creating the original design group and 
their continued use as a panel of consultants could be very important 
for the field* Also, the efforts aimed at linking the more conceptual 
Researchers with the strong field survey organizations for work in 
this area (as suggested above) could act to add considerable capacity 
to the educational R/D&I system. 
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2. Monitoring ' 

The issue of monitoring as a critical requirement for R/D&I 
system building and orchestration was frequently noted in our general 
analysis. We have also already touched upon this in our review of 
this case»» While monitoring was not an initial or explicit purposs 
of the study, it is evident that to make use of its findings (if 
these turned out to be of value) for the design of future dissemiia- 
tion and development strategies would demand an ongoing monitoring 
process. Additionally, the type of data that would be collected wculd 
provide a basis for the design of a system to monitor the existence: 
and functioning of a potentially very significant phenomenon in the 
R/D6cl system. Hence, in orchestrating the survey design, execution 
of its phases, etc., NIE should make explicit its desire to have tta 
efforts lend themselves to these monitoring purposes. 

3. Environmental Impact 

It may be that by paying explicit attention to the value of 
self-help efforts, and presenting in a positive light NIE's desite 
to See a synergistic interweaving of internally and externally- 
generated innovations »• that positive consequences could be generated 
for the Agency and its R&D efforts. These might involve increase 
in legitimacy ("We do not assume that all wisdom comes to you from the 
outside") and in the political climate ("We are really seeking to involve 
quality Ipcal capacity"). 

C. Internal Nl^ and Other Non-Procurement Behaviors 

The central theme of the proposed NIE strategy with respect 
to this study was that frequently recommended in this analysis- 
orchestration. The key to the strategy was the careful orchestration 
of various types of participants so that they could play their optzaal 
roles at the right stages, the building-in. of key checkpoints or 
milestones, the encouragement of desired collaborations, etc. 
Clearly this would call for NIE personnel capable of performing such 
tasks, as well as the synthesis of ideas for the study as we recozcend- 

ed after stage 1. / 

We also noted the need for cooperation and coramunication across 
a number of groups within NIE that could (or should) have been inter- 
ested in the study. Further, NIE might consider with which other 
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agencies (e.g., OE) It should be cooperating in the conduct of this 
study and in what might grow out of it. 

• Finally, there nay be other (essentially) non-procurement actions 
that would be seen as contributing to the various purposes of the 
study. One such example might be the holding of a session at AERA on 
the topic, or a special conference. There may be others. 

6. Likely Impacts 

We have suggested throughout this scenario what we believe might be 
a number of th^ likely Impacts of this alternatively designed procurement. 
We summarize tlUse here: 

a) Ther^ woW seem to be a strong possibility of highly creative 
thinking in the survey design stage. 

b) Consequently, the long-term data base and monitoring system 

. evolving out of this survey program would likely be approgriately 
oriented toward: developing increased understanding of the R/D&I 
system in education; and identifying useful leverage points for 
policy interventions. 

c) Several syStem-Uuilding impacts seem likely: developing coramli- 
nication linkages around the strong Research talent of the rele- 
vant Research areas (e.g.. "invisible college" mechanisms); 
providing additional support for, and facilitating the quality 
work of strong Research organizations working ""n relevant Research 
areas; affecting the long-term Research agendas of these strong 
organizations and increasing the potential for synergy across the 
work of these various organizations (and between their agendas and 
NIE's); and possibly over the long run developing linkages among 
the self-help Districts and strengthening their capabilities «s key 
participants in the R/D&I system. 

d) NIE's close working relationship to the field in carrying out this 
procurement and related activities is likely to strengthen NIE's 
position vis ^ vis the field and enhance the Agency's image as an 
Integral and key part of the field facilitating its development. 

c) Other positive environmental impacts would also seem likely. SEA and 
l£A personnel are likely to view with favor NIE initiatives that 
recognize their Innovative potential and accord them respected status 
as key organizations in the R/D&I system. Given the generally strong 
influence of education interest groups on Congress and the public, this 
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would seem to suggest important side-benefits for the Institute's 
image, legitimacy, and long-terra stability. } 

f) i.IE*s position as the lead agency for Research and Development in 
education is likely to be strengthened by cross-agency coordination 
of the kind suggested here. Given the critical interrelationships 
between OE and NIE programs, it would seem clear that collaborative/ 
coordinating strategies between ,0E and NIE (as well as other agencies) 
should have substantial long-term payoff. 

g) For NIE, one^ot the most significant organizational impacts of this 
alternatively designed procurement may be the promofton of internal 
communication across NIE "divisions, the development of synergy 
across divisional program lin^s, and the emergence of a systems 
perspective across the Agency — with all that implies for the con- 
sideration of interaction/ irtfolio effects, l^uilding multi-purposes 
(especially system building) into procurements) and designing pro- 
curements in relation to ^ variety of non-procurement actions that 
in the long run may have greater consequence for the system than any 
individual procurement or set of procurements. 

' The procurement ^s Initially conceived might clearly >ave accomplished 
much that could b^^ viewed as highly beneficial, and i€ is not our purpose to 
find fault with procurements made in this manner.. Still, we believe that the 
benefits likely to accrue from the redesigned procurement are substantially 
greater. Therefore, we recommend this approach to the Institute and the 
Council^^ for their consideration. 
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II. UNSOLICITED PROPOSALS; A PROPOSED Fj^AMEWORK FOR PESIGN OF NATIONAL 
PROGRAM EVALUATIONS 
1. Description of the Procurement* 

Among the unsolicited proposals received by NIE in mi^ 1976 was one 
particularly interesting description of a framework for design! of national 
program evaluations . The approach was designed to provide the \klnds of pro* 
cess and impact data that could be expected to meet the information needs of 
decision makers at different levels of government as well as progr&ji managers 
and staff. The approach permitted some degree of local variability in de- 
signs and emphases while at the same time enabling. data to be aggregated or 
disaggregated to meet different decision needs. What made the proposal seem 
particularly intriguing was the comprehensiveness with which program charac- 
teristics and implementation conditions were treated in the scheme outlined. 

The Research organization that submitted the proposal was requesting 
funds for„ elaboration and subsequent testing of the framework across several 
national programs varying in key program and/or implementation characteiistics. • 

The proposal appeared to be relevant to the program concerns of sev- 
eral NIE divisions and was therefore reviewed by each of ".hese in turn. In 
some cases, program personnel were not sufficiently interested in the proposal 
to be willing to fund it out of their own allocations. In one case, there was 
substantial interest in funding the proposal but available funds of that di- 
vision for the giverx program year had already been committed. The leadership 
cf that division suggested that perhaps the needed funds might be secured from 
the budget of the Director's Office where there was available a small reserve 
fund for potentially significant funding opportunities of this kind. 

The Director' Office requested that the proposal be reviewed by sev- 
eral leading members ot the Evaluation Research and Research Design fields. 



*While the case is presented to read as though it describes an actual NIE pro- . 
curement, it Is In fact hypothetical. Also, the authors imply no po-^ition on 
the substantive content of the procurement. 
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Favorable comments were received from most o£ these reviewer^ with only 
minor points suggested here and there for rethinking. These points, along 
with some additipnal considerations and possible reservations, raised by a 
knowledgeable NIE staffer, were then forwarded to the proposing Research 
organization. After some discussions, a mutually agreeable Research Grant 
scope of work was drafted, calling for full development of the proposed 
scheme and field testing of its utility and effectiveness at specified sites 
of designated federally-funded national educational programs carried out under 

NIE or OE auspices. 

NIE staff h.-?.d previously met with OE personnel to advise OE of the 
proposed grant, suggest and work out the details of the needed coordination 
between agencies, and secure OE approval and cooperation in the use of 
OE-funded programs and OE-supported program sites in the field test. OE was 
enthusiastic about the grant since it had been supporting related work on the 
development of a set of somewhat standardized models for ESEA Title I evalua- 
tions. The work supported by OE was similar in its orientation toward making 
possible data aggregation or disaggregation as needed to meet different in- 
formation xieeds. But the OE-sUpported work was less powerful in" its lack of 
attention to implementation conditions (a weakness recognized by the OZ con- 
tractor and taken into account in their longer-term plans). 

2. Impacts 

In considering what the likely impacts^ of this procurement might be, 
the following seems reasonable: 

a) The work might well produce a significant breakthrough in 
resolving some of the methodological dilemmas of the Evalua- 
tion Research function at this point in its development. 
Its positive reception from leaders of the field might be 
^viewed as encouraging. However, since breakthroughs 
tend to be relatively rare and Research of this kind in- 
volves a .considerable amount of uncertainty, it is also 
quite possible (and perhaps even more probable) that dif- ' 
ficulties encountered a^ong the way will reveal unantici- 
pated problems' in the proposed framework, weakening its 
utility for widescale application. If such is the case, 
then the substantive Research output might better be 
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classified as an important (or possibly unimportant) 
part of the accumulating knowledge/ techno logy base of 
the Evaluation Rei^^^ir ch function, and not as a key 
breakthrough in the field's methodologica] -levelopment . 

b) NIE's involvement of the field in evaluation of the 
proposal is likely to be well .received, at least by the 
part of the field included in fhe evaluation process. 
Given the NI-E strategy of including the field's leader- 
ship in t\i{s process, the payoff in generating positive 
field affect is likely to be substantial (unless of 
course 'the field is undergoing a period of change or is 
divid:ed into different "schools of thought" and NXE'a 
selection of advisers takes into account only one of 
several groups vying for influence on the direction of 
the field's development). 

c) NIE-OE coordination of efforts is likely to strengthen 
the links between these two key education agencies. 

3. Evaluation of the Procurement and its Impacts. Actual and P otential 

In evaluating the procurement. Agency personnel must weigh actual 
(or likely) impacts of a procurement as initially formulated with possible 
desired impacts. The question to consider here is: How might such a 
procurement be designed to better achieve Agency purports, or to achieve 
a broader range of Agency purposes? 

At the very least this question requires insuring some clarity on 
Agency purposes: What in fact are the Agency purposes that are viewed as 
relevant to this procurement? Might a somewhat different view of the 
Agency role and its requirements for procurement policies produce a 
different or an enlarged picture of the relevant purposes? 

At the very least, NIE's role is conceived as that of a funding 
agency which can facilitate quality work in the field through choices as 
to which work it funds. Therefore, the funsolicited proposal would likely 
have been evaluated in accord with some/of the criteria we suggested in 
our discussion of the Research functi^: Does the proposal fall within a 
Research area with which we have decided to work? If it does, can the 
proposal be judged to demonstrate the level of state-of-the-art creativity 
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and Excellence that calls for support. Jf it falls outside a Research 
area that we have decided to work with, but is of such excellent quality 
that NIE should in some way or other be part of its sponsorship, how 
might NIE work with other agencies to insure and to coordinate its sup- 
port?) Does NIE have the kinds of people in-house who can work with 
the proposing Researcher (s)? Which Researcher (s) and which organiza- 
tion(s) will be supported if the proposal is funded — a "star" of the 
field? an exciting new talent? an existing center of excellence with 
the needed minimum, critical tnass of relevant talent? or, an institu- 
tional setting lacking in the organizational resources needed to ade- 
quately carry out the proposal or build long-term system capacity? 

Once we start considering this latter group of questions, we have 
moved contiderably beyond a conception of NIE's role as simply a funding 
agency seeking to identify individual proposals to fund. Instead, NIE's 
role is viewed in terms of broader, more system-oriented respoi^sibilities. 

One of our basic contenti.ons in our discuss on of the Research 
function was that the latent, long-term, system-building potential of any 
Research procurement was likely to be far more consequential than the 
manifest substantive piifpose of the procurement. If this argument is 
accepted, and if we define the relevant Research area as Evaluation 
Research methodology, ttiei^ several aspects of the Evaluation Research 
function as it exists in tltie education sector today become relevant to 
evaluating the proposed approach to making this procurement: 

a) The knowledge/ technology base of the field has been 
undergoing extensive development in recent years. 

b) There is a clearly visible Evaluation Research cDnou- 
nity, with its own leadership, its own channels of 
communication, its developing standards of quality 
work, etc. 

c) Though the numbers of Evaluation Researchers and in- 
stitutional bases for Evaluation Research are large, 
the amount of first-rate talent is somewhat limited and 
is distributed unevenly across the various types of per- 
forinep organizations in the field. 

d) There is a huge federal investment in Evaluation Research 
especially if one aggregates Evaluation Research 
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expenditures across agendles. 
e) There is substantial foundation interest (the Russell 
Sage Foundation) in the development of Evaluation Re- 
isearch methodology. ^ 
Therefore, it seems reasonable to consider how much greater the po- 
tential impact of this unsolicited proposal might be if it was used as a 
base on which to build a broader-scale, longer-term, staged, multi-purpose 
program, coordinated across governmental agencies and internally within 
NIE across divisions and program boundary lines. 

4. Possible Alternatives 

A. Work with the Proposing Researcher in an Interactive/Coll aborative Mode 
If NIE had tong In-house staff with the needed^ methodological skills 

(and there would seem to be some staffers with these talents at present), 
the Agency could work with the proposing Researcher to strengthen the pro- 
posal, relate it to other work going on in the field and other relevant in- 
itiatives funded by other ^agencies (e.g., OE) . In-house personnel could 
possibly; facilitate information exchanges between the proposing Researcher 
and others in the field; develop meeded syntheses and critical reviews 
mapping the area and the state of the art, useful for stimulating field com- 
munication about the proposed framework (and the ongoing work to elaborate 
and test it) j make data available for use in elaborating or testing the 
framework's utility prior to field workf etc. 

B. Using the Proposal as a Basis for Developing Close Working Relatlon- 

wlth the Strong Talent in the Field 
NIE might consider convening a continuing seminar or panel to discuss 
the implicaitions of the proposed (and subsequently the ongoing) work, to 
assess its quality, and. to consider what it does or does not suggest about 
directions for the methodological development of the field as well as the 
work's practical applications in national program evaluations. This kind 
of ongoing seminar might become a wedge for design of alternative ap- 
proaches ( or components of a single unified approach), each carried out 
by different, participants in the seminar, alone or working together 
collaboratively. 
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C» Cross-Agency/Funder Coordination 

NIE'mlght play a leadership role in contacting other agencies or 
funders} (e.g., the Russell Sage Foundation) to stimulate their interest 
in this proposal as well as the possibilities of identifying synergistic 
add-ons or points of relevance to work sponsored by these other funders. 
Possibilities for pooling of resources night be a particular point of 
focus In these interactions. 

D. Internal NIE 

All NIE project officers might be asked to consider their prograja 
evaluation needs and how these might relate to the proposed framework. 
Such discussions — In-house, with the proposing Researcher, and with the 
long-term advisory panel of leaders of the Evaluation Research field — 
would seem to have the potential to stimulate cross-program communication 
and synergy in the development and Interrelatiou of Evaluation Research 
designs for NIE programs. 

5. Summary Description of the Alternative Procurement and Related Activities* 

A> Staging and Field/Agency Relations 

The procurement and related activities could be thought of as having 
the following possible elements: 

1) collaborative/interactive NIE-Researcher development of 
the proposal in its final form; 

2) NIE staff development of state of the art syntheses and 
wtitlques that relate the proposed framework to the ex- 
isuing technology base of the field; 

3) convening of a long-term^ on-going seminar of leaders of 
the Evaluation Research and Research Design communities 
to assess the validity, significance, and implications 

of the ongoing work — as well as to facilitate the devel- 
opment of an ''invisible college*' mechanism; 



We wish to remind the reader of the hypothetical character of both the 
process described and our neutral position as to the substantive content. 
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A) design of alternative models, or system components of 
a unified model, with different members of the seminar 
group working on pieces of this, individually or col- 
laboratively J 

5) testing of the initially proposed framework (as elab- 
orated), and possibly alternative models, through 
reanalysis of available Evaluation Research data sets; 

6) field testing of these frameworks /models in a small 
sample of selected program sites; 

7) large-scale field testing across varying programs 
nationwide; 

8) consideration of utility of developed models at interim 
points in the testing procedure.; 

/ 9) convening conferences of Evaluation Researchers and pro- 

j gram personnel from varying national educational programs 

/ to consider the imp"* ications of this work, 

sjveral of these stages might be clustered together, but what is impor- 
tant iiB that each stage suggests critical questions that will need answers 
beforfe it will be obvious a" to whether it is justifiable to proceed. 

B. Multi-Purposes 

The manifest purpose of this procurement — i.e., to elaborate and test 
the proposed framework — appears to be of .some consequence in and of itself. 
But in addition, other latent purposes could be achieved as well if the pro- 
curement was considered in system-oriented terms. Wc have noted several 
ways in which system-building purpo^re's could be achieved (e.g., facilitating 
communication among, and providing additional support for the work of, the 
strong Research talent and Research organizations in the field). In addition 
positive environmental impact might accrus from involvement of the program 
personnel m the assessment of the new framework and its implications after 
the field tndt ras'ilts are available fot considetation. 

6. Likely Impacts 

a) NIE's close working relationship to the field in evaluating and 
then expanding -.he scope of this proposal is likely to strengthen 
ViE's position vis a vJs the field and enhance the Agency's image 
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as an Integral and key part of the field facilitating Its 

\devel o pmen t • 
) As we have noted) several system-bulldlng Impacts seem likely: 
developing communication linkages among the strong talent In the 
Evaluation Research and Ilesearch Design fields (e.g.» "invisible 
college" mechanisms); providing additional support for and facil- 
itating the quality work of strong Research organizations working 
in relevant Research areas; and affecting the long-term Research 
agendas of these strong organizations and Increasing the potential 
for synergy across the work of these various organizations (and 
between their agendas and NlE*s)» 

c) NIE*8 position as the lead agency for Research and Development in 
education is likely to be strengthened by cross-agency coordination 
of the kind suggested here. 

d) Increase4: communication among NIE program officers (considering the 
implications of this proposal to their programs) Is likely to 
strengthen the development of stronger program evaluation? that con- 
sider interaction effects across programs as well as impacts at- 
tributable to discrete programs. 

Our conclusion here Is similar to our statement In the previous scenario. 
The procurement as initially conceived might clearly have accomplished much 
that could be viewed as highly beneficial. And, clearly > much that we have 
suggested is similar to courses of action frequently followed by the Institute 
What may be different here, and what may explain any additional benefits 
likely to accrue from the expanded description of the rethought procurement 
and non-procurement actions, Is the emphasis on simultaneity and Interaction 
among these occasionally considered options. We therefore recommend this 
kind of thinking to the Institute and the Council for consideration. 
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I« FRAMEWORK! A SYSTEMS VIEW OF EDUCATIONAL R/D&I 

1. Developing an Analytical Framework for a New Field 

Institutionalized Research, Development and Innovation (R/D&I) in 
education is little more than a decade old — yet the R/D&l system 
capacity (as we can assess it now) , our under standiiig o£ the 
system and our ability to manage it have increased significantly. 

We are now In a position to begin to de elop and refine, over 
the next few years, an analytical framework and a relatively 
unobtrusive monitoring system (for data gathering) with which 
we may assess the educational R/D&I system in terms both of 
progress made to date and of what might reasonably be expected 
in the near term And - ^ger terra future. Such an assessment 
would provide the basis for annual or periodic reviews of the 
educational R/D&I system. 

The analytical framework and the monitoring system for such 
H'ssessment could be developed from our growing knowledge of 
R/D&I in other sectors and of the conditions pertinent to the 

education sector in particular. 

\ 

In this brief overview report, we will suggest in broad terms 
what such a framework might look like; what should be the basis 
for assessment In the current and succeeding periods; in these 
terms, but based on the incomplete and tentative evidence and 
impressionistic judgements available at this time, what is the 
status of key elements In the system and what can be expected 
in the near and longer terms; and finally what major needs 
require consideration in formulating federal policy and program 
initiatives. 

2. Educational R/D&I as a System 

It is important to make explicit a key premise that underlies 

ERIC 



the proposed framework; namely that the Institutions and personnel 
Involved in the production and utilization o£ educational R/D&cl 
outputs form a "system" and not Just an unconnected "configuration" 
of entities. Apceptance o this premise does not deny that there can 
be and often Is only a weak linkage or integration between institu- 
tions which S|hould be more closely related and whose goals might 
show more cohiirence. Nor do we imply any monolithic, centralized 
network. But there are very significant Implications for long-term 
planning and monitoring and for the development of initiatives by 
a federal agency that do come from such a "system" perspective; 
i.e.» of a leadership that can give meaning to an expansion and 
maturation of educational R/D&I capacity. ^ 

Most particularly, the perspective permits a proper concern for 
capacity building and system maturation. It also permits a 
recognition of those fundamental or generic characteristics com- 
mon to all R/D&I systems and allows us to learn from their diffi- 
cult experiences in growing to maturity. An informed jolicy per- 
spective for educational R/D&I must be based on an understanding 
both of those generic Issues and of the special characteristics of 
of the educational context. 

3. Stages of Development 

R/D&I systems characteristically go throtigh various stages of 
growth and development — from the uncertainties and insecurity 
of birth and early years; to a period of striving to establish 
themselves and their legitimacy; to a more mature period in 
wftich their functions, institutions and linkages are well estab- 
lished. 

In thbs early stages, one finds unrealistically high expectations 
and over optimistic forecasts concerning high quality results - 
to be delivered quickly and implemented on a widespread basis. 
Since in reality the development of an R/D&I system (and its outputs) 
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takes much longer and requires raucl]^ more investment than is usually 
recognized at the time, one soon al£o finds frustration and a 
tendency to "overreact" on the part of R/D6'.I system personnel, 
funders axid sponsors • 

Thus, it becomes important to assess the value of investments 
to date in terms of whether or not a' strong basis for the future 
and future returns has be#n developed — not in terras of sunk 
costs^ disappointments and unrealistic early expectations. 

4. An Expanding View of Educational R/P&I 

.Over the past two decades, federal funding policies have refected 
an increasingly broadening perspective of what constitutes an 
educational R/D6el system. Initially, funding was charnelled 
primarily to Research; then increasingly to Development (e.g.r 
design and development of curriculum, instructional packages, 
etcOj to Evaluation Research in the 1960's; and, more recently, 
^.ncreasihgly to Policy Research. ^ 

Most recently, there has been a dramatic increase in funding 
support for functions which link R5cD outputs with utilization 
in school ^ystems i.e., the Dissemination function and assis-- 
tance for the Implementation/Utilization functions. Indeed, 
support for Utilization activities represents 43% of educational 
R/D&I funding in both FY75 and FY77 (projected). 

In addition to a broadening perspective as to which R/D&I functions 
to support (as noted above) ^ there has developed an expanded view 
of (1) the institutions in which varJrous R/D&I functions are per- 
formed (thus, for example , we find substantial federal funds being 
channelled to User system institutions t^ develop User system 
capacity for innovation); and (2) the type of institutions with 
which NIE may establish special relationships (o.g.^with academic 
and private aector institutions as well as with the federally 
created labs and centers). 
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It» ASSESSMEOT OF THE DEVELOPMENT OF SYSTEM CAPABILITY OVER THE 
DECADES 

\, T^e Bases for Assessment of the Educational R/P&I System 

v 

An effectively functioning mature R/D$«I system must have: 

1) a network of stable institutions of minimum critical 
mass, properly attuned to their 'various functions 
(Research, Development, Dissemination, etc.)> and 
appropriately linked to each other and to Users; 

2) qualified personnel in sufficient numbers, prop rly dis- 
tributed and focused on appropriate programs; 

3) visibility and legitimacy among the various R/D&I system 
stakeholders ; . « , 

4) adequate and stable levels of funding; "^o^ 

5) a stiong knowledge and technology base; 

6) system mana§,ers, decision makers, policy makers who 
hdve relevant management and policy tr fining and 
skills. • 

Only when these conditions exist can one expect the R/D6el system 
CO have the capacity to provide a meaningful flow of high 
quality outputs (which could improve educational practice) that 
are adopted and used effectively by a significant number of IJseri 

The current state of the educational R/D&I system's capacity 
and quality of activity may be assessed, but the assessment 
must also involve: 

1) a comparison with the state of Educational R/D&I in the 

past, and 

2) the potential for future R/D&I system development and 
maturation. 

2. The Historical Context 



The challenge that has faced us over the past two decades has 
been one of virtually creating an educational R/D&I system 



de nova* R&D work did exist — but hot a cohesive R/D&I system 
properly linked to Users • 

Specifically 9 two decades ago one would generally find: 
1) scattered r(^Bearchers (u...dtly individuals or sma'll 

teams in academic settings studying schools and students^ 
nonetheless having few strong linkages to User system 
personnel); 

^2) a sm*ll number of large curriculum improvement projects in 
selected subject aiceas (againj staffed largely by univer- 
sity scholars) ; * 

a considerable amount of in-hoase, practice -based devel- 
opment of curriculum (by school .system personnel); 

private companies providing textbooks j materials and 
equipment and a very few small Research corporations* 

None of these tended to have significant discernible impact on the 
User system or on the accumulation of a knowledge or technology 
base directly appllcable^o^the solution of Uder system needs and 
problems ♦ This is no^ surprising when we consider that R/D&I 
generally lacked : 

1) strong linkage* between Researchers, Developers, Producers, 
Disseminators and Users — or even ambng personnel within 
a specific functional area <e*g*: among Researchers); 

2) significant leadership or toordination either from the f 3 ' \d 
or from any national level organization* 

3> The Institutional Base , 

We can identify some significant changes over the past two decades 
in the inatitutional base of the educational R/d&I, system. For 
one thing, the mission oriented organized R/D&I mode has replaced 




the Indlvldual/small team mode as the dominant mode o£ R&D» F6r 
another, there are now some 1500-3000 R/D&I organizations (precise 
figures hp functional/organizational category must await survey 
work now in progress)* These include (at the very least): 

1) the various educational organizations crei^ted through 
federal initiative and funding (e.g.: the regional 
laboratories, R6cD centers, ERIC clearinghouses, materials 
centers, etc#), 

2) R&D capacity in several hundred academic and private sector 
Research or R&D institutions /organizational units (including 
private publishers, equipment companies^ etc.); 

3) R/D&I capacity (mostly newly created) in every SEA, in ISAs 

i 

and even in several hundred of the larger LEAs» 

Most of this capacity is relatively new (even on the SEA/ISA/LEA 
levels), having been created or , expanded with federal funding* 

The impact that federal funding/procurement policy can have on an 
R/D&I system is well illustrated by the dramatic development pf the 
private sector noted above (which has been receiving a quite sub- 
stantial percentage of the federal educational R/D&I dollar) — 
compared to (and at the expense of ) the academic sector (which 
in recent years has received a relatively small percentage — in marked 
contrast to earlier years). To illustrate, in FY 1971, only 21% of 
DHEW evaluation awards were to the academic sector ) while 74% went 
to the private sector (29% to non-profit and 45% to for-profit 
corporations) ^ this despite evidence from at least one study 
suggesting that evaluations done by the academic sector have 
tended (on the whole) to be subs^J^ntially superior to evaluations 
done by the private Sector. 

Another point to note is that a Su/ill numbe' of institutions 

fewer than 50 — are receiving the bulk of NIE R/D&I funding. Does 



this mean that these fewer than 50 essentially represent the 
ejftent of the R/D&I system's High quality institutional base? 
Or are there other high quality institutions which (for what- 
ever reaspn) have not found NIE funding activities attractive? 
And if so, how many? We currently have no data with which to 
answer these questio' though preparatory work is already 
underway at NIE for a future set of studies to analyze dis- 
tribution of institutional capability in relation to various 
kinds of R/D&X activities. 

A third point to note is that there has been some signifi- 
cant instability within this expanding R/D&I institutional 
base. Institutions or new organizational units have emerged -- 
only to disappear a few years later for lack of funding or mar- 
kets for their services. Only 7 of the original 20 regional 

it 

laboratories remain today, and only 12 of the 13-21 centers. 
However, we must also note, (admittedly impressionistic) evi- 
dence suggests that: (1) the expansion rate has been signi- 
ficantly greater than the contraction; and (2) there appears 
to have been some degree of "leveling off" of the "shakeout" 
of existing R/D&I institutions (as seen, e.g., in the strength- 
ened political position of the labs and centers) . We would 
also note that this observed historical pattern with the R/D6el 
system institutional base is precisely what one would expect 
in introductory and late introductory stages of an R/D&I 
system's development — i.e., a substantial degree of insti- 
tutionalization has occurred (with some losses) , and a healthier 
degree of career stability has become possible within even the 
newer of these institutions. Given the predominant federal 
role in funding educational R/D&I activity, future decisions 



*The types of organizations included under the "center" rubric 
varies in different discussions. 



about the level, direction, and form of funding to these in- 
stitutions (which we know to be very responsive to such funding"'' 
patterns) will be determining as to their future. 

4. Personal Base 

Although we generally lack precise data about the educational 
R/D&I system's personnel base, the following general estimates 
and analyses can be made: 

1) In comparison to the raid-'eOs, the educational R/D&I 
personnel base has doubled (perhaps tripled). The 
best estimate was that the R/D&I system personnel base 
in 1969 totalled about 4,000 persons. In 1974, sev- 
eral estimates suggest a mean figure of about 10,000 
persons (estimates ranged from 8-12,000, and higher 

or lower estimates can be found, depending on one's 
definition of an educational R/D&I system). 

2) Most of that work force is represented by Researchers 
and by Development personnel. 

3) There would appear to be a particularly inadequate num- 
ber of personnel to carry out linlige roles in Dissemin- 
ation/Feed Forward and support roles for Need Identifi- 
cation, Implementation/utilization especially if these 
functions ave expanded over the next few years to the 
degree suggested by various programs now in the planning 
or discussion stages. 

4) Most educational R/D&I personnel have an educational or 
psychology training background (roughly 90% in 1965 and 
80% in 1974) and university or school system work back- 
grounds. Given the interdisciplinary nature of educa- 



tional R/D&I personnel base. In this respect, the above figures 
represent limited progress, but perhaps less than one might have 
expected. 

I 

When the current survey of R/D&I organizations is completed and the 
personnel data are analyzed, we will have a relatively good basis for 
determining the precise size of the personnel base, hov7 it is dis- 
tributed by work setting, functional specialties, and representation 
of minorities and women. 

Precise data on the participation of women and minorities in the 
educational R/D&I work force must await the results of the current 
crganizatlonal survey. of R/D&I performers. (Based on such indicators 
as AERA membership, it would now appear that neither women nor 
minorities — except Orientals — are present in the personnel base* 
in proportion to their numbers in the population — though the level 
of participation of women is significant ~ 28% of AERA membership.) 

Until additional contemplated special studies of the personnel base 
are undertaken, we will lack the needed data we need to understand the - 
sources of that work force (by background, training, career history) 
or the incentives, expectations, and aspirations likely to affect the 
success of policy initiatives to expand that base, affect its diver- 
sity, recruit personnel for new system roles, or stimulate considera* 
tion of mid-career shifts in patterns of" functioning. 

Educational R/D&I planners have raised serious concerns about im'- 
balances and weaknesses in the system's personnel base; e.g« : 

1. an inadequate number of specially trained personnel in 

the User-oriented functions (dissemination, laiplementation/ 
Utilization) ; 

2. limited availability of strong training programs specifically 
designed for functional specialties (except for Basic Research); 

3. a far too limited number of first-rate talent being 
attracted to the field. 

None of the corrective strategies attempted pver the past decade have 
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been wholly satisfactory. As a case in point, the substantial sup- 
port received for Research training programs in the '60s seems to 
have contributed substantially to the training (and probably to some 
extent to the recruitment) of about 1,000 current educational R/D&I 
personnel. However, it is not entirely clear that the training 
provided In these somewhat conventional Research training programs 
was the kind of training needed to meet the needs of li/D&I functioning, 
and the large scale, hxgh priority, federally- funded programs have 
been discontinued. Although some subsequent training programs have 
received federal support (e.g.: new programs to train change agents 
and other linkage personnel), the scale and level of funding of current 
programs would not be adequate for either rapid expansion of the 
system's personnel base or near-term correction of the Imbalances 
In the current personnel base acrosf. functional specialties. 

Our knowledge of other R/D&I systems suggests : that : 

1. ) The rate at which the personnel base can be expanded 

varies among R/D&I system functions. 

2. ) In Research (and to a lesser extent, Development), the 

rate is dependent on the number and size of the existing 
centers of excellence (which alone can provide the training) 
and is a long t6rra process. 

3. ) For the linkage functions (Dissemination and to a lesser 

extent Development), training- programs can be developed at 
relatively modest levels of funding and personnel trained 
within a relatively short time frame. However, training 
■in these functions will be constrained by (1) rates and 
levels at which Users can reasonably absorb their outputs 
and (2) the relative lack of codification in the knowledge/ 
technology bases. 
A.) Thus, merely investing dollars in training is not always 
wise or effective. 

v. 

5. Outputs 'f )'i 

Educational R/D&I activity has produced a substantial number of out- 



puts over the past decade or two. Some 776 of a much larger number 
of practice-oriented outputs f^^veloped with OE or NIE funding over 
the pagit decade are listed in the. 1976 Catalog of NIE-Sponaored 
Educational Products. Clearly, a listing of the total number of out- 
puts during the past two decades which have been produced by all 
institutions within the R/D&I system from all funding sources would 
bs many times larger. 

At this time, there are few data-based statements that can be made 
about overall quantity or quality of these outputs. There has 
been considerable criticism phrased in rather general terms about the 
poor quality of educational R/D&I outputs, but this tends to be 
Impressionistic or based on examination of a relatively small pro- 
portion of what has been produced. 

At ..he fame time, the system has always produced at least a few 
outputs? of outstanding quality and widely reputed excellence — the 
NSF science curriculum improvement projects of the late '50s and 
early '60s (PSSC^ CHEM, BSCS, the School Math Study Group); Indi- 
vidually Prescribed Instruction; Sesame Ltreet; etc.. In an ef- 
fort to identify and make better known some of the other high 
quality outputs of the system, projects to Identify exemplary products 
of R&D efforts have brought increased attention to at least 30 such 
outputs. Examples of a few of these products with extensive utili- 
zatlon histories are: 

- Sullivan Reading Program (programmed readers) , reportedly 
being used bv more than 5 million chLxdren 

- Science Curriculum Improvement Study (fundamental concepts/ 
elementary school science) , reported to have been used by 
more than 1 million students 

Science — A Process Approach (elementar)' school science 
taught through processes of observing, measuring, classifying, 
predicting and inferring) , estimated to be used to teach mil- 
lions of stnxtemtff 
~ Kindergarten Program, or First Year Communication Skills 

Program (basic skills of English language eommunlcatlon) , used 
by about 250,000 students • 

- Man: A Course of Study (social science curriculum for grades 
5-7), reported to have bep" used by at least 200,000 students 
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SEAs (with an additional Impetus from federal support) have been 
Identifying, packaging and disseminating potentially high quality 
^'exemplary practices" developed by. local school personnel for their 
own use* As yet, it is too early to judge whether the exemplary 
practice concept will be a major source of innovation for the educa- 
tional system or an Illusory will o* the wisp, more hopeful than 
practical or workable in reality — as one of Its critics charge. 

There has also been an attempt to provide comparative evaluative in- 
formation about alternative products, practices and packages to meet 
given needs* Agattn, it will take some time before we will be In a 
position to assess the quality or utility of these outputs • 

6* Linkages to Educational Practice 

Clearly, whatever might be said about the quantity or quality, effective- 
ness or utilization of outputs of the educational R/D&I system over 
the past two decades, it would be fruitless to contend that the in** 
vestment in the educational R/D&I system to date has produced sub^ 
stantial, widespread Impact on educational practice* There have been 
some clearly outstanding outputs produced by the system that have been 
well received and have reasonably extensive utilization histories* 
Some firm linkages have appeared between some of the stronger Develop- 
ment organizations (e*g,: some of the labs, centers, and private sec- 
tor firms) and the school systems who have been using their products. 
On the whole, however, there has been rather minimal Integration 
between R&D, on the one hand, and educational practice, on the other* 
Neither the findings of educational Research nor the products of 
specialized R&D organizations or the highly publicized innovative 
strategies of recent decades have generally been found to have any dis- 
cernible impact on educational practice; where they have been found, 
they tended more often than not to be emasculated into "more of the 
same old thing." 

School systems that are highly innovative have been described as using 
local resources to develop innovations designed only for local con- 
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sumption. Consequently* a vast body ^ of practice-based innovation 
fails to get documented or disseminated. 

Innovation in education, then, where it does exist, is not being 
managed with maximal efficiency for widespread impact on the educa- 
tional enterprise in this country. But clearly, the picture described 
above is what one would expect to find for a relatively young R/D&I 
system in the early stages of its development — i.e., sporadic 
examples of significant, impactful knowledge production/knowledge 
utilization linkages amid a general picture of much less substantial 
achievement . 



One of thje most promising trends in recent years has been the increased 
attention given by funders and policy makers to the development of 
effective linkage mechanisms. For example: the substantial share of 
federal R&D' funding allocated to Dissemination, Implementation, and 
Utilization activities (43X of federal educational R/D&I funding in' 
FY 1975^; the increasing flow of federal R/D&I funding to User 
system Institutions (SEAs, ISAs, LEAs) ; new types of R/D&I activity 
being supported with federal funding (e.g.: State Dissemination 
Capacity Building Grants and LEA local problera-solvitig projects and 
capacity building accompanied by documentation and analysis of the 
processes used .so that they can be packaged and disseminated to other 
LEAs); etc. 

Precisely how much linkage capacity now exists in plape, how it is 
distributed, and how adequately the various existing mechanisms link 
Producers and Users are questions we cannot yet answer on the basis 
of existing data pr even data surveys now being planned Hare 
specialized studies of Producers, Users, and linkers will be needed. 

7. Funding ' 

In FY 1975, total funding for educational R/D&I in this country, from 
all sources, fell somewhere between $504 Trillion and $565 million 
(depending on what is included or excluded in a given estimate) , with 
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$576 million the most likely figure. Of this total sum, approximately 
82%, i.e., $470 million (with a possible lower bound of $430 million 
and a possible upper bound of $520 million) , came from various federal 
departments or agencies. The remaining sources of educational R/D&I 
funding Include: State funds, $40 million ($30 million to $60 million); 
local government funds, $4 million ($2 million to $10 million); private 
foundations, $57 million ($57 million to $65 illion) ; and other private 
sector sources., possibly (but here estimation is especially difficult) 
$5 million ($3 million ta $75 million) • 

By comparison, the level o2 funding estimated to have been invested in 
educational R/D&X by all &ources in FY 1968 was $250 million. Data 
for the yrevious decade or so ate 1J:SS adequate, but some idea rf the 
majgnitude of the increase in funding levels can be seen in data from 
USOEy the largest single source of funds for educational R/D&I acti- 
vities; USOE appropriations for research and training rose from $1 mil- 
lion in 1957 to a little ovar $100 million in 1969. The level of 
funding available fron; federal sources has levelled off somewhat since 
1966 (in terms of rate of growth) when the period of the most rapid 
expansion of the system^ s funding base ended. 

Federal funding for educational R/D&I activity is provided by a sub- 
stantial number of federal departments and agencies. Different data 
sources vary in the way they define relevant R/D&I activity and the 
departments and agencies from whom budget data are collected. The best 
recent set of data, gathered by a National Academy of Sciences study 
group, provides data figures from 21 departments agencies (counting 
four HEW units separately). Fourteen of these departments and agencies 
provided $438 million of the estimated $470 million or so estimated to 
have been Invested by the federal government in educational R/D&I in 
FY 1975. The largest source of funding came from the Education Division 
of HEW ($335 million or 73%): of this $335 million, $294 million came 
from the Office of Education > $74 million from the National Institute 
of Education. Substantial sums also came from the Department of Agri- 
culture ($56 million) , the National Science Foundation ($40 million) , 
and the National Institutes of Health ($38 million). Quite a large 
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number of other federal agencies provided smaller amounts of funding. 

After the current survey of educational R&D performer organizations 
Is completed, we will be In a better position than now to estimate the 
relative size of funding allocations by functional areas of R/D&I acti- 
vity » and to determine to what extent the available resources are appor- 
tioned in a manner so ae to provide the appropriate balance between 
functions (taking Into account the overall stage of development of the 
I R/D&I system and any necessary corrective actions that may be needed to 
redress previously out of balance conditions) . 

The NAS data provide useful information about how £i%deral funding for 
educational R/D&I Is divided among groupings of functional areas* The 
FY 1975 data indicate that knowledge production (research, evaluation, 
and statistical activities) have received the smallest share of federal 
funds (17%) , while 40% was allocated to applications formulations (materials 
development, policy formulation, demonstrations, and social experiments), 
and 43% to Utilization activities (dissemination and implementation 
. activities). Comparison with FY 1977 budget data indicates that know- 
ledge production is receiving somewhat more attention (23%, up from 17%), 
while applicat^f^n formulations are receiving less (34%, down from 40%) 
and support for Utilization activities has remained stable (at 43%) . 

Comparison of 1975 and 1977 budget figures suggest at least one area of 
concern: when the rate of inflation is taken into account the federal 
allocations suggest a 14% decline in real terms financial resources 
available to educational R/D&I. 

8> Knowledge/Technology Base 

There are two kinds of knowledge relevant to the building of the edu- 
cational R/D&I system: (1) substantive knowledge/technology relevant to 
specific system functions (needed by functional specialists) and (2) 
knowledge about operative aspects of the system: what exists, what 
policies/strategies do or do not work (needed by system managers and 
policy makers) . 

er|c 
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In the educational R/D&I system, the knowledge/ technology base Is Weak 
in both respects — probably substantially weaker than for most other 
existing R/D&I systems. This Is understandable! the educational R/D&I 
-system is relatively new; it exists in an emotional, value-laden, 
political context; and it is multi-discipliuary; it has had a relatively 
low level o£ funding. 

Nonetheless, so long as the system's knowledge/ technology base remains 
weak, we vill nand to find poor quality outputs, which will (in turn) tend 
to (1) produce and/or reinforce a negative environmental climate for the 
system, and (2) inhibit the flow of funds, personnel and other inputs 
irto the R/D&^ system — inputs needed for system maturation. 

Given the above understanding of these overall weaknesses, we must also 
take note that relatively substantial progress has been made in the past 
' ten to twenty years — and that there are hopeful signs' for the next 
decade. To illustrate, one of the most useful (though admittedly "rough") 
indicators of the state of development of the knowledge/technology base 
needed by functional spec-' alls ts is .the availability of handbooks and other 
syntheses of the relevant information — i.e., some degree of maturation 
must exist before these can begin to appear. 

Using this criterion, we find that educational Research and Evaluation 
Research have the beginnings of a modest knowledge/technology base, while 
Dissemination and Development remain relatively weak. 

Additionally, there have been efforts to develop knowledge of the opera- 
tional aspects of the system (e.g.: NIE's R&D System Supp'jrt Program). 

Thus, there are signs that the system's knowledge 'technology base will 
improve in the years ahead. However, we must be aware that this involves 
a slow, system building process it is not done easily, nor will it be 
done overnight. 



5.; 
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the quality and stability of support and funding. It is highly 
dependent on its roots ia its fundamental disciplines* Its out- 
puts are knowledge and stimulation and it is only generally in the 
long term that we can as. ;iss its practical contribution. And, ' 
given its inherent uncertainties it becomes hazardous to attempt 
to predict the areas in which such outcomes will occur. But with- 
' out it the well of the new thinking frequently runs dry. It is 
therefc^re vital that a healthy and mature R/D&I system will have 
^ developed and maintained a substantial high quality Basic Research 

component. But such a component \ cannot be built quickly. The 
rate at which quality Basic Research can be expanded is limited by 
the size and quality of i^ta existing centers of excellenc^# To. 
pump more funding into this endeavor than such centers can usefully 
absorb can only lead to waste and disappointment. ,Future growth 
is (and will be) limited by past investments in creating and sup- 
porting a central core of Basic Research having many centers of 
excellence of minimum critical mass and larger. The major problem 
in educational Basic Research has been in the very weakness of this 
central core. 

Assessment of the Basic Research function will need to be based on: 

1) The size and quality (based on the reputation of institu- 
tions and personnel) of the central core of the Basic 
Research function — most specifically on the size, growth 
and stability patterns of ' identified centers of excel- 
lence. An important indicator will be the ability to at- 
tract and hold top flight researchers. 

2) The number of new centers of excellence seeded and taking 
* root over successive (rolling) 3-5 year periods. 

3) A measure of the supportiveness of the climate - in terms 
of funding growth and stability over several year periods. 
Measures of the quality of the linkage to and reputation 
of Basic Research in education and its more fundamental 
root disciplines (e.g.: psychology, sociology, etc.), 
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Over long (10-20 year) time spans, an assessment of major 
substantive contributions to .knowledge coming from educa- 
^ional Basic Research. 

B, Current Status and Expectations 

Basic Research In education is generally, to be found in two types 
of 8e(;tlng8. That located in schools of education (frequently 
in such disciplines as Educational Psychology and Sociology) 
J)oasts few centers of excellence and much mediocrity. A dif- 
ferent picture emerges from viewing the research carried on in 
discipline based university departments such as Psychology and 
Sociology. Excellence and valuable contributions to knowledge 
are to be found, but what has been lacking here has been a 
primary and continuous commitment to education. Tlie interdis-- 
cipllnary character of educational Research has added to the 
diffuseness by making communications and Information retrieval 
(from the lar^") variety of publication sources) very difficult. 
Altogether > this has added up to an educational Basic Research 
community that has been to date unstable and amorphous. It 
makes system building in this area a major requirement for NIE 
efforts and a critical consideration in funding programs. 

At the same time, the general climate for Basic Research in 
education for other (especially social) areas of Basic 
Resea ch has been far from supportive. This negative climate 
has been particularly intense for education which has been hard 
put to point to more than a handful of significant developments 
that are. traceable to Basic Research. The low prestige with the 
general public and with Congress and the associated unreliable 
funding have made it hard to attract strong talent and this has 
acted as a major constraint on building the central core. As 
regards funding, it is vital to note that NIE has been a 
relatively minor contributor to the total funding going to Basic 
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Ressarch, especially in comparison with such an agency as NIH, 
Thus, for example, in FY 1975 in the area of early childhood and 
adolescent education 5% of total federal funding for Basic Research 
came from NIE while 187> came from PKS/NIH. 

C. Key Needs . 

The key need for the educational Basic Research function is a 
consistent, continuous, stable process of system building. This 
vould include: 

1. identifying existing centers of excellence f* - -^ 

2. facilitating the establishment of additicffal centers of 
excellence; 

3» facilitating the growth of these centers, existing and 
new; 

A. facilitating improved information exchange and retrieval 

mechanisms ; 
5. providing stable, long term funding. 

2. Problem-Focused Research 

> 

\. Assessment Basts n 

It Is important to keep in mind thajt Problem-Focused Research is_ 
Research and shares with Basic Research a high 13vg1 of uncertain- 
ty ind unpredictability. Thus^ Researchers in particular treat 
Problem-Fo?cused Research in a Basic Research mode. But it is. also 
targeted Research. Thus, Funders and Users often assume it to have 
the level of certainty and shortness of time line more appro- 
prii;tely associated with Development. This decep tiveness and the 
^consequent inherent tension makes Problem-Focused Research "sub- 
ject to considerable instability, misdirection and mlsmanagemen,t , 
and coixsequent misdirected assessment: 

Researchers frequently redefine and bend Problem-Focused Research 



into Basic Research modes. In particular, they often attempt 
to undertrke projects on smaller scales than are required by the 
nature of the problems, which often require the efforts of large- 
scale interdisciplinary and empirically based team programs. 
This syndrome is often combined with attempts to oversell the 
timing, probability and impae.t of outcomes in order to obtain 
funding. This often succeeds with funders simply because 
Problem-Focused Research projects do appear to have practical, 
attainable outcomes. All of this creates ax environm^.nu uhat 
tends to be unattractive to many of the best Reseat 

On the other side, Users and Thunders, having been persuadsu to 
fund such prot,rams becaupfe of these very expectations of near- 
time benefits, become frustrated by not only the lack of delivery 
but also by the shifting targets, time and cost patterns which 
are inherent in the uncertain Research process. 

Another important dimension of this tension lies in the problem 
of Need Identification. On the one hand, the objective is to •. 
work on important and timely problems that require solution, aitd 
this tends to be the prime inducement for the Users and FuuJers. 
On the other hand, a Researcher is required to maintain the 
criteria of researchability — criteria that often significantly 
limit the utility of the project from the User perspective. 
This as well as the previously mentioned problems of tension 
become magnified when one recognizes that the cost and scale of 
Problem- Fo< • -d Research tend to run orders of magnitude higher 
than what Ir typical of Basic Research. 



• Assessnent must therefore be based on ^'udgements o^: 

1) The quality and appropriateness of the institutions 
performing this function: j 




Are they attracting and keeping top quality ap- 
plied researchers? 

Are their programs and projects considered 

to be of high quality, important to practice and 

on truly researchable problems? 

2) Wliether^ Problem Focused Research Is emerging as a 
definable entity, differentiated from Basic Research 
and Development* 

3) After a time lag that reflects several years of sus- 
tained system building, an evaluation of the rate and 
impact of outputs* 

4) The climate for Problem-Focused Research in terms of 
both support patterns and receptivity to its outputs* 

\ 

B, Current Status and Expectations 

Most of the Research that is carried on in education appears to 
be what night loosely be defined as the Problem-Focused type, 
much of it unfunded and small-scale* The volume of studies 
produced may Indeed be large — but being of this small-scale, 
scattered and fragmented quality, these have been subject to many 
questions of quality* It is evident that there Is substantial 
lack of differentiation In education between what can truly be 
classified as Research and various other activities (e.g*: demon- 
stratlon projects, social bookkeeping, etc.); great weakness* in 
defining researchable problems; considerable fuzziuess in dif- 
ferentiating Problem-Focused Research from Basic Research and 
Development; and the previously mentioned tendency to oversell 
such projects. 

Proy,em- Focused Research in education is largely carried on in 
two types of institutional settings: universities and large- 
scale R&D institutions in the privates and quasi-public sectors.*^ 

Where this work has gone on in universities, there has been a 
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tendency to perform Problem-Focused Research in a Basic Research 
node. This is not surprising given the socialization and prior 
training of university Researchers and the social and publica- 
tion pressures under which they operate. Generally, univer- 
sities find it difficult to assembJ-a the minimum critical mass 
of ;;ffort needed to undertake large-scale Problem-Focused 
Research projects. As a consequence, they have tended to scale 
such projects down and/or to assemble ad hoc teams that lack 
long-range stability. With this has come the unfortunate ten- 
dency for Researchers to move in and out of this part of the 
field which has mitigated against system-building requirements. 

Large scale R&D organisations should have been, and to some 
degree aave been, more suitable sites for such programs. 
However, two important probleus have limited their potential 
success. Firstly, most of Iheae R&D organizations have not been 
able to promise a stable Cc^ieer path to Researchers, thereby 
greatly liiui tine, t^eir ability to attract and hold first-rate 
Researchers. Second]/, federal funding practices in the late 
60s shifted the character of many of these institutions away 
from Prsblem-Focused Research i.'nd reshaped them into Develop- 
ment or ,ani2-.uions in accorc^ance with federal priorities at 
tl.»t time tor product-centered impact strategies. 

Ac. a consequence of the above conditions, education has In fact 
seen very little Problem-Focused Research. Therefore, this has 
to be seen as an area that needs to be put together at this 
tinie in its ovm terms and not be thought of as a form of ad- 
vanced Devnlopmeat or downstream Basic Research. 

A number of other problems in educational Problem- Focused Re- 
search deserve mention. The climate for such Research has been 
perhaps even more negative than that described above for Basic 
Research. This has been so precisely because it seemed to hold 
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out more promise of impact and raised expectations than could 
have been satisfied — given the inherent time frame and the 
weak state of the area. Relatedly, Need Identification, which 
had been Researcher-driven up through the mid- '60s, became 
system-driven by Users and Funders in an overreaction to this 
state of affairs. As with Basic Research, funding has been 
relatively limited, receiving under 5 per cent of total federal 
FY 1975 funding for R&D in early childhood and adolescent 
education. It is interesting to note that NIE's rule (at 
least recently) has been a more significant factor in funding 
Problem-Focused Research than was the case we observed vrith 
respect to Basic Research. (In FY 1975, of all the federal 
R&D funding in the applied Research category in early child- 
hood and adolescent education, ^5% was from NIE alone.) 

C. Key Needs 

Problem-Focused Research in education, thpn, must be seen in 
a system-building mode. 

1. It will be essential to locate thos^ centers of ex- 
cellence capable of performing large-scale Problem- 
Focused Research. 

2. Such institutions will need to be provided with the 
kind of long-ten table- funding that will permit 
them to attract and retain top-flight staffs of 
Researchers. 

3. It will also be vital for the lead educational 
funding agencies to help practitioners and Lhe 
Congress understand the nature and requirements 
of Problem-Focused Research; to understand that 
project selection requires the determination of 
what is researchable as well as what is important; 
to recognize that the present lack of capacity 
demands a period of institution-building btafore 
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the promise of the area can begin to be fulfilled; 
and that such institution-building will require an 
ongoing and long-term commitment. 

Development 
A. Assessment Basis 

Critical in the assessment of the Development component of an 
R/D&I system is the recognition of the centrality of its 
linkages to the User, to Pro<luction and to the state of art 
in Development. Development has a relatively more predictable 
and shorter time horizon process as compared to the Research 
functions. It aims to convei't knowledge into User ready products, 
products which may (or may not) need to pass through a distinct 
Production phase before they can be dis_seminated or distributed. 
With the linkage to the User being so critical, so is the 
requirement for Need Identification — a step that is diffi- 
cult to perform, but one that must be done well and often in 
an ongoing (manner during the Development process (where com- 
plete Identification is not feasible ~ as in many areas of social 
development) if the product selected for development is to be 
on target. 

Development is also highly dependent on the quality of its 
linkage to the state of the art and on the skills and motivation 
with which products are desigaed- so as to be capable of produc - 
tion and dissemination. This determines the effectiveness and 
viability of the product. It depends on the quality and ap- 
propriateness to the task of the institutions involved and the 
knowledge base upon which they must depend. 

Since Development is frequently carried out in specialized 
Developrjent organizations it is highly dependent on the quality 
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of such institutions and their personnel and most particularly 
on their experience # It is important to differentiate the 
concept of excellence in Development from that used in Research* 
In Development excellence is measured by being cost/effective, 
timely and opportunistic* With such criteria, experience 
(individual and organizational) and thoroughness (ability to 
to the whole Job) are often more important than brilliance. 
Where Development is carried out in a User setting, then 
assessment must be concerned with the extent of wider dissemina^ 
tion* 

Thus, the critical bases for assessment are: 

1) quality of linkages to: 

Users 

Production 

Development state of the art* 
Measures of such linkages are hard to define and 
obtain, depending as they do upon quality, frequency 
and form of interaction. They will likely be quali- 
tative in nature, and "observable" more in their 
absence il^ terms of problems generated, than in their 
presence • \ 

A specific manifestation of this linkage^ will be in 
the quality of Need Identification, to be measured 
indirectly by the relevance of Development outputs 
for practice, and by the scope and effectiveness of 
feed forward activity from Users to Developers. 

2) Number and effectiveness of large scale Development 
organiz«*tions. Effectiveness here would be measura-' 
ble in teirms of extent of adopted products and some 
qualitative assessment of impact (actual and poten- 
tial). 
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3) Extent of dissemination of practice based develop- 
ments. 

A) The number and quality of products developed from the 
whole R/D&I system and their overall (portfolio) 
effect. Quality would again have to be measured by 
usage based criteria although the locus of quality 
control would be a design variable. 

« 

B. I Current Status and Expectations 

Educational Development is plagued by a weak knowledge base. 
The quality of information is poor, very little has been 
codified. As such, quality control is a central requirement, 
which has only recently begun to receive serious attention 
but which Is still relatively poorly developed and its 
enforcement a matter of some dif fuseness as to locus of re- 
sponsibility. With limited ability to depend on quality 
control in the field, funders and program managers may need , 
to build quality control checkpoints into staged Development 
procurements, a procedure that demands closer Agency/Field 
involvement and orchestration than has been typical to date. 

Two modes of Development which predominate in educational 
Development at this time are the "specialized Development 
organization" and "practice based" Development. 

At its best the specialized Development organization repre- 
sents a strong element in educational R&D system capacity. 
There are a few such well staffed and experienced Development 
organizations and their existence is an important indicator 
of tjie system building that has gone on. More of ten,'.-how- 
ever, the institutions and personnel Involved In Development 
do not come up to these required standards. Ev^en the best 

i 

of these organizations tend to suffer from isc^lation from 

ERIC 3(r; 



-28- 



practice, making Dissemination and Implementation problema- 
tical. This may be one of the causes of the limited utili- 

« 

zation of R&D based products, a shortcoming that is tend;tng to 
threaten the viability of this type of institution. There 
may be a critical need in Development not to increase the 
level of effort overall (there is an inventory op more than 
half a million R&D products that are available for sorting, 
tailoring, packaging and dissemination) but rather for a 
shift of emphasis so as to build up more of the strong high 
quality Development organizations with whom the government 
can contract and. to ensure their closer linkage to practice. 

The second basic mode of Development in education (practice 
based) does not suffer (obviously) from poor linkage to the 
User. It does suffer, however, from inefficiency, lack of 
sophisticated skills, poor documentation of J.t8 achievements, 
difficulties with packaging, and from enormous problems in 
achieving wider dissemination and diffusion. The verdict is 
not in as to whether this mode can become a source for wider 
application (beyond the local development site). Meanwhile, 
further research on this mode is required as well as govern- 
ment efforts to supplement local capabilities (possibly 
through renting strategies). 

In the area of project selection the. emphasis to date has been 
on a project-by-project selection process. Missing has been 
the capacity in the system to consider critical portfolio ef- 
fects. These could involve decisions to target and concentrate 
Development programs so as to achieve synergistic benefits, . 
staging and sequencing strategies that minimize User disruption 
and uncertainty, cooperative ventures across agencies, etc. 
Particularly important may be the need to develop skills in 
commercialization so as to make better decisions with respect to 
what to place with which elements of the private sector and when. 
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C. Key Needs 

1. Build up the explicit designing-in of quality control 
functions into funded Development programs - possibly 
including staged programs with quality control check- 
points . 

2. Shift support emphasis to favor those high quality ' 
specialized Development organizations that show a pat- 
tern of being responsive to practice needs and to- tech- ^ 
nical opportunity. 

3. Research programs to determine the most cost effective 
Jaya of identifying and disseminating practice based 
Development products. y' 

^ 4. Investigation and trial of mechai:{isms to supplement 
local Developmental capacity. 

5. Development of program plannirjfg and project selection 
methods by federal funding agencies that give explicit 
consideration to portfolio effects. 

6. Explicit programs designed to achieve inter-agency 
cooperation for Development activities. 

7. Study of and experimentation with strategies designed to 
improve the government funded Development organization- 
to-commercial firm interface — including development of 
criteria for what should be handled how and by whom. 

4. Dissfcimination 

A. Assessment Basis 

The function of Dissemination is critical to the entire R/D&I 
system. It is, In essence, a linkage process which "connects" 
knowledge producers with knowledge users. Thus, the R/D&I Dis- 
semination system must provioe for mechanisms which: can deter- 
mine what is available; can sort out the "good" from the "bad"; 
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will allow Users to Identify and obtain the particular 
product" which are relevant to their needs; as needed, can 
•'tailor" products to fit User needs; can motivate Users to ^ 
•'try** a product; Insures effective User Implementation and ^ 
utilization. 

Assessment must be made in terms of capacity to achieve and 
s uccess in each of the above requirements. Overall we would 
wish to know this with respect to: 

(1) Extent and quality of "reach" into U-^er systems 
(e.g»: number being reached, the extent of repeat 
utilization of Dissemination services, and User satis- 
faction with such service) 

(2) Levels of User awareness and trial of R&D products 
(existence, character, and evaluative), 

(3) Contribution to Implementation and Utj(lization of R&D 
products. Since this depends on such other factors as 
number and quality of products available. User skills 
and receptivity etc., the Dissemination function can 
only be assessed as a contributor to the prcjcess. 

This must of necessity be a qualitative evaluation. 

I 

(A) The existence of a well developed and coopetative net- 
work of Dissemination mechanisms giving coverage across 
the nation and to the variety of Users to be four1. 

B> Current Status and Expectations 

In education, we find a number of problems and barriers to Dis- 
semination. There are an enormous number of ^^sers (some 19,000 
school districts — plus teachers, etc.), among whom there is 
wide diversity and variety as to philosophy. Interests, perceived 
needs, etc.. Innovations make demands on the time of school per- 
sonnel (a very practical matter) and generally require "people 
change** — factv>rs which can lead to resistance to innd'^atlon. 
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Addi.tlonally, at least two major factors have tende.d to 
create a vexy poor climate for Dissemination in education: 
(1) a lack of Implementation/Utilization support to the User; 
and (2) the perception t.\at R&D products havf* been inferior 
to existing User-developed products. 
"\ 

In education, there has been a considerable amount of activity 
that has been called Dissemi.;ation , and a large number and 
variety of organ: l^•*ons are involved in some ki|nd of Dis- 
semination, and a ^rge number md variety of organizations 
are involved in some \ind of Dissemination — b't much of this 
has be^i fragmented and scattered (e.g.: "add-ons" to De^'clop- 
ment projects; successful but separate and discrete Dissemina- 
tion systems for specific categorical programs). 

In the past decade, federal efforts in Dissemination have 
focused upon* the ERi!'^ system and the state of Dissemination 
capacity building grants. Again, howe-'er, these are not ef- 
fectively linked pr orchestrated into a national "system", 
and receptivity with Users varies greatly. Further, there is 
not yet a well developed personnel base of trained Dissemina- 
tion spp.ciallf ts. Several federally funded programs have been 
developed in rece,.it years for ^.raining Dissemination and 
Utiiizat-ion speci'.alists , but Dissemination mechanisms are ex- 
panding far more rapidly and creating a far greater demand for 
trained personnel than these programs could even hope to keep 
up with. 

rrom an overview perspective, then, the need is for: 

1. orchestration of educational R/D6.I Dissemination from a 
tot al system per sp ec live ; 

2. in the short term, facilitating the work existing Dis- 
semination mechanisms and "filling" critical "^;aps"; 

3. in the long-term, providing for overall system building 
(this calls for policies and strategies which are proac- 



•tlve, not passive or reactive, and which are based on a 
knowledge of what does. and does not in fact exist) ; and 
A. balancing short and long-teriii needs, 

♦ 

More specifically, federal policies and strategies must focus 
upon: 

1.: quality control; 

2f mechanisms that can optimize product /Ms semination/User 
; "fits"; 

3. providing Users with alternative channels of access to 

the available resource base (a "mixed strategy" approach). 

Federal policy will need either to expand the Dissemination 
training capability or slow the rate of Dissemination system 
expansion, keeping in mind the level and rate at whiih Users 
can absorb hew input once a Dissemination system Is established 
— a factor which Is of pritical Importance. 

Federal policy w^ll need to provide for systematic, ongoing 
xnonitoring of the Dissemination function. 

5. Evaluation Research 

A . As s essme nt Basis 

In or» r to understand and assess Evaluation Research In the 
education context, it Is important to be aware that Evaluation 
Research Is a multif aceted R/D&I function with some potential 
problems and conflicts 'Inherent within the function. It is 
both Evaluation (a Knowledge Utilization function) and Research 
(a Kr.owleCjge Production function) . Its primary purpose Is to 
reduce the level of decis-fon uncertainty for decision makers. 

y 

Thus, it has a clear, specific focus (which is the problem de- 
fined by the decision maker) ; seeks sufficient (not "ultimate") 
truth; must be "timely", within short time constraints. Yet it 
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is also Research, to be done by Researchers who are generally^ 
used to l^ger time frames, to leas, clear "targets", to 
defining their own "problems", etc. There are potential con- 
Slictsof needs and interests of policy makers /funutrs on the 
one hand and of program administrators on the other. Evaluar 
tlon Research may validly be needed to provide both ongoing, 
corrective evaluation data (Formative Evaluation) aftd "end- 
result" (Suramative evaluation) --' yet the Research design for 
one conflicts with the Research design for the other. Evalu- 
ation Research must often evaluate programs in differing real 
life' contexts and yet provide data which can be aggregated. 

Specifically, assessments would be Isased on the following: 

1. in general, those bases outlined foi^ the Basic and Problem- 
Focused Research functions; 

2'^ the extent to which identified c^snter^s of excellence and 
identified competent Evaluation -Researchers are used in 
government sponsored Evaluation Research; and 

3. the extent to which recommendations (stated or implied) of 

^ Evaluation Research are acted upon. 

B. Current Status and Expectations 

Evaluation Research in the social science context *(of which edu- 
cation Is a part) was strongly affected by the proliferation of 
large-scale federally funded social programs with attendai^ 
requirements for reporting and analysis of program effective- 
ness. 

Of all the R/D&I functions in the field of education, Evalua- 
tion Research has experienced the most rap^.d and extensive devel 
opment in the last ten to twelve years, both in terms of sup- 
port through federal funding and in terms of an increasingly 
self-consciou^ "identity" as a separate function. Though sub- 
stantial progress has been made, the field still lacks an 



adequate theoretical base; i|astruinentatlon Is in a|rudl- 
mentary state of developmentjj basic conceptual and methodo- 
logical dllemnas remain unresolved there are relatively few 
persons well trained in Evaluation Research (though many 

♦nore than existed ten to twelve years ago). ' 

\ ■ 

... \ 
The bulk of federal Evaluation Research f uj|iding has gone to 
the private sector, with relatively little going to the unlver- 
• sity setting — yet some evidence suggests that quality Evalua^ 
tlon Reaeai^ch comes from the acadea''*c community. , 

■ . ■ • y ■■ V • 

Evaluation Research in education must be done in, a highly 
value-laden, politicarl context — with a multiplicity of group 
and Interests having reason to use, dispute, manipulate or bury 
either pjsitlve or negative findings. Valid issues such as who 
controls the design and outcome of Evaluation Research must be 
settled within this context — and must J>e settled\in the early 
(design) stages of the process. 

Educational Evaluation Research must 4eal with problems associ- 
ated with the nature of the eduratlonal context itself : thel size 
and variety of the ed^<5atlon field may result in plknned or un- 
planned design variations; multiple levels of government may need 
different data — yet such potentially different data must some- 
how be aggregated. 

C J Key Needs 

System building per se is not a primary concern there are 
various Evaluation Resear"^ organizations already in the field* 
However, funding raSy need to be 'focused niv-^r^ on the academic 
sector — perhaps even upon a selected set of univajrsities and 
university-related organizations . 
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Because of the critical na^ture of the Design and Data Analysis 
stages of Evaluation ReaearchI the beat talent in the field 
Wst be utillaed. This intplles a relatively high degree of 
agency control over the selection process-. In the Data Col- 
lection stage, control over selection, is not a critical Issue. 
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Policy decisions must be made early in any Evaluation Research 

process concerning: 

I" 1. Whose evaluation information needs are to be met 
(policymakers, program administrators, various 
leve^-s of govei?xmient , etc;); 
2. The relative roles of decision makers and Evaluation 
Researchers In defining the problem from a specific 
/ Evaluation Research project; ^ 

" • 3. Whether or not data shall be made public — and when. 



In a word, EvAj-uation Research must be done well — ^and it 
probably will not be done well without NIE leadership in 
orchestrating the many complexities and potential conflicts 
involved. 




/ 



CONCLUSION 



. have throughout this report noted weaknesses in the educational ^ 
^./D&I system. It is Irjiportant now to re-emphasize that we have also 
noted uhat ^^jiat we have found would be generally what one would 
expect "O have found within a relatively young R/D&I system. There 
has been, progress, and there are signA p.f the beginnings of a transi- 
tion from the Introductory stages pfc-^ystem development . V 



Thus, as we note 



.1 at the outset, the current state of th^ educational 
RyD&I system tnust^e assessed in teri^s of where it has be^^n and where 
it now has ,the potential to go not in terms of unrealistic expecta 
tiofls about "progress and output to date." 
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with this perspective In mind, we can see the last two. decades as 
a period of bome Important- achievements In the creating and 
building of the educational R/D&I SfHem In the United States. 
As compared to twenty years ago: 

''-^ There are today some 1500-3000 organl7atlnns (academic, 

i ■ ' , * 

private, and public) which have R/D&I capacity — • most of 

this capacity being relatively new and being largvsly the 

result of federal funding. 

, 2. The personnel ba^ie has doubled (perhaps tripled) — from 

around 4000 In 1969 to 8>12,000 In 1974. Most o£^ this 

work f ord\3 is represented by Research and Developn&ent per 

sonnel. \ 
^, ihe educational R/D&I system has produced a substantial 
number of outputs. Some of these have been outstanding 
quality and of a widely reported excellence — products 
from R&D organizations and exemplary products' which' lave 
f been identified, generalized and widely dissemihated, 
4; Sorte linkages have been developed between some of the 
strong Development organizations and the scb .1 systems 
who have been using their products. 

5. USOE funding for research and training rose from $1 mil- 
lion in 1957 to a little over $100 million in 1969. 
Total Funding (federal and other) for educational R/D&I 
has risen from $250 million in 1968 to a(n escimated $576 

^ million in 1975 — though when inflation ^L^ considered, 

1977 federal funding appears to be about below 1975 
levels. 

6. Since some degree of maturity of a knowledge/ techroiogy 
, base is necessary to allow its codification into handr" 

- books and other syntheses ,y<7e may infer some beginnings 
of educational R/D&I systdra maturity from the increasing 
lev^l of availability-^ such handuooks/synthesea. 



Aa the R/l&I systbm has matured, Inevitably some of the functions 
have developed and/or tean supported more than other'3. It will be 
Important to maintain a* "balance" betweer these various functions 
of the educational R/D&I system. This balance mus^ take into ac- 
count for each function ; 

1» the time period needed to produce significant outputs; 

2. the Impact each function has on the other functions; • 

3. the level of funding needed both to maintain a balance and 
to maintain the basic integrity of the personnel and in- 
stltutlonal base within each function; 

A. what currently does/does not exist within each function 
(In terms of outputs and of the Institutional personnel, 
knowledge and technology bases) • 

As we looked across the various functions within the educational 

R/D&I system^ we noted the following: 

1, The central core of Institutions and personnel In the Basic 
Research fiinction seems to be particularly weak (in terms 
of persons comitted to and devoting the bulk of their 
^reers to educatlLonal Basic Research) • This points both 
to a need for support and a limitation on the rate at 
which this Base may be developed, '^he linpact of Basic 
Research must be viewed over a lon^-term (not a short-^term) 
time frame. 

?.• There is a sizeable community of educational Researchers 
working within the Problem-Focui^d Research function. 
However » tmich of what is being done is small-scale and 
scattered, while what is often needed requires a large- 
scale approach. 

3t In the Development function, wc find a sizeable number of 
Development products available and there are son^e trong 
Development organizations. — though relatively few in 
comparison to other fields • The Development function has 
received a large portion of federal funding over the ^ ast 
decade. 



4. In tile Dissemination function, we find a generally weak 
function characterized by fnuitiple fragmented activities. 

. There is a need for system building — but care must be 
taken not to build a system that would overwhelm User 
capabilities to absorb what Is disseminated. 

5. In the Evaluation Research function^ we find the begin- 
nings of a significant Evaluation Research community, 
though we must note (a) that the bulk of federal funding 
has gone to the private sector and (b) there is need to ; 
determine the relative potential capabilities of the 
academic and private sectors for high quality Evaluation 
Research* 

With the above in mind Uhd considering the limited federal funding 
available), the following general format for federal funding of the 
educational R/D&I system would be relevant: 

1, Provide moderate funding for Basic Research for long-term 
systems building purposes and to insure ongoing stability . 
within the personnel ant* institutional bases. Funding 
should not be premised on short-term, "quick" outputs. 
2» Since larre scale Problem-Focused Research has been ne- 
glected, provide major funding here for system building — 
but not at a level greater than tha capacity of the func- 
tlon to absorb productively. 
3. Since there are now a sizeable nuqjber of Development pro- 
ducts available, reduce current Development funding — to 
that level necepsary to maintain rhe c •: Jffng high quality 
centers. 

A. Since Dissemination has been so fragmented, direct signifi- 
cant funding to Dissemination — but not so much as to 
build a syst.em that would overwhelm Userd. At this time, 
quality con::rol, sorting and technical service would lil aly 
need to be a part of the function. 

5, Provide moderate funding for Evaluation Research for system 
building purposes. At this time, some emphasis needs to be 
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placed on clarifying the role and potential capabilities 
(for' high quality work) of the private and academic sec- 
tors. 



In sunoary, the period of the last two decades has been an Important 
era of Initial ays tyi building for the educational R/D&I system. 
There reajjfitn problems, weaknesses, critical gaps to he filled, 
balanced to be achieved as one would expect to find In a relative- 
ly young R/D&I system. These identified needs ^become the fdcus for 
system building and rebuilding for the next transitional phase for 
the next five to tii years. In this period, it will be yital.to 
provide continuity, stability and security in 'ord4r that the educa- 
tional R/D&I syst^em can take root, grow and develop maturity. Only 
In these ways can we hope to develop a maturing educational R/D&I 
system which can have significant impact on the educational system 

In the UAl ted States. . ^ \. ,.->h 

s 
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The National Institute of Educatlonl(NIE) is a lead mission- 
oriented agency for educational resei^rch and development in 
the United States. It is a lead agency both because it was 
created for. this purpose by the Congtfess and because the scope 
of its mission encompasses all aspects of the educational R&D 
process. While NIE thus has a "lead" ^role, the nature of its 
mission and leadership can only be undiprst^od in the context 
of the many other parties concerned witfh educational R6tD: 
various governmental agencies at the federal,, regional, state 
and local levels; quasi -governmental an| non-govemirental 
organizations; the operational educatioi^al system (plrimary, 
secondary and post-secondary educational'i institutions rnd their 
administrator /teacher/student personnel kt the local and state 
levels); and, indeed^ the various individ^ual publics whose 
(often conflicting) interests, needs and ljemands both impact 
and are the "end result" concern of educational R&D. 

One part of NIE's overall responsibility is fundamental research 
relevant to education. . In order to fulfill this part of its 
responsibility, NIE has sought the advice And counsel of the 
National Academy of Sciences (NAS) . This »^dvice and counsel 
been presented in "Fundamental Re'search and the Process 
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of Education" (1977), a report of the NfS Committee on Funda- 
mental Research Relevant to EducatioV. In response to th^ NAS 
report, the ProgrLOJ Conimtttee of l;he National Council on Educa-^/^ 
tlonal Research (NCER) has presented a series of t)ollcy recom- 
mendations In Its own report to NCER. 

The NAS report emphaslases the Importance of _ fundamental research 
relevant to education and has very correctly and :apab.ly focused • 
on many critical aspects and requirements of fundamental research. 
We are In strong agreement with the report's emphasis on the 
Importanct! of fundamental research relevant to education as we 
have stated In one of our earlier analyses for l^E (Radnor, Splvak 
and liofler 1976). At the same time, a careful reading of the NAS 
report reveals some very critical deficiencies. These deficiencies 
could mislead NIE Into developing policies and strategies which 
would be Inappropriate and dyfutic tlonal In ternis both of NIE b 
overall responsibilities and of the basic lnt;ent of the NAS com- 
mittee: to strengthen fut^iamental research relevant to^ducatlou. 
Indeed, as we shall discuss in this analysis, such a result has 
already occurred in the form of the NCER Program Committee recom- • 
mendatlons regarding funding and research coitmunlty Involvement 
recommendations /which appear to have compounded the ""imitations 
and potential-fotL-error of the NAS report. 

From an overview perspective, the major deficiencies of the NAS' 
report (and the NCER Program Committee Reports) include the ^ 
following: 

1) NIE has lead responsibilities for all aspects of 

educational R£eD, not just for fund... ental research. 
While the NAS Committee was charged to examine only 
fundamental research, NCER/NIE deliberations cannot 
consider fundamental research apart from its othi-r , 
areas of responsibilities. The NAS report does mention 



f 

this Issue -- but only marginally , la effect, the NAS 
report is what one might expect from a committee 
already biased towards fundamental research a bias 
which the NAS committee openly and forthrlghtly states • 
In a similar vein, we would expect a committee com- 
posed of educational development , arsonnel to make a 
similar analysis and set of recommendations about the 
importance of (and the need to strengthen) develop- 
ment relevant to education. NCER/NIE have the 
responsibility to balance policies and strategies 
across all aspects of educational R6eD a balanc^N^t- 
present in the NAS report and potentially threatened \ 
by the Irecomroendations of the NCER Program Conmittee. 

The NAS report fails to examine a number of issues 
about the educational R&D conjCext -- Issues which 
significantly impact policy/ strategy deliberations 
for strengthening fundamental research relevant to 
education. Among these educational RSd) context issues 
would be: the current institutional/personnel base for 
education- re levant fundamental research (number and 
quality; interest and commitment; capability to use 
productively- increased levels of funding — and what 
levels); the rate at which the personnel/ institutional 
base could be "built up"; the derivative, multi- 
disciplinary nature of- educational R&D; that education 
is also a practice-based field where experience-based 
knowledge may be a vital component of the process 
by which new concepts and insights are developed; the 
relative immaturity of the educational R&D field. 

♦ 

What is the role of NIE as the "lead" agency in light 
of: the many institutions concerned with/involved in 
educational R&D; the variety of disciplines in which 
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fuadamental research may be relevant to education; the 
fact that NIE is only one of many federal agencies 
which provide funding for fundamental research relevant 
^to education? While the NAS report does suggesj: some . 
valid and important roles for NIE relating to funda- 
mental research, there are significant issues, roles 
and role options which are' not considered by the 
NAS report. • 

While both the NAS and IVCER Program Committee reports 
enqphasize the need to strengthen fundamental research 
relevant to education and make certain recommendations, 
neither provides adequate rationales or criteria to 
guide the policy and strategy dali^erations of KGER/NIS. 

The NCER Program Committee has recomflftnded that 20% of 
NIE'is funding be allocated for the support of fundamental 
research relevant to education by 1979, and at least 307b 
by 1985. (No specific figures or dates were mentioned 
in the recommendations of the NAS report.) This NCER 
Program Committee recommendation is seriously deficient 
in a number of ways. No rationale is provided for the 
selection of these specific target percentages and 
dates. No analysis is provided about the impact of such 
a funding policy on fundamental research. Neither is 
analysis provided about the Impact of such a funding 
policy on the needs of (and NIE's responsibilities for) 
other aspects of educational R&D (for example: need 
identification, "applied research, and dissemination 
areas which many believe to be even> weaker than funda- 
mental research in* the education context) . 

The NAS report fails to deal with the distinctions and 
relationships between fundamental research on the one 



hand and applied research on the other. For example, the 
report at times seems to be making a clear distinction 
btetween fundamental and" applied research; yet at other 
times, the report seems to use the terms almost inter- 
changeably. As a poignant case in point, the report uses 
a study by Connally (1977)* as anr example of the- type of 
important fundamental research needed (p. 33), when in 
fact, this^jas an applied research study on an applied 
problem (project selection and monitoring) in a highly 
applied context at NASA.** As a further illustration, 
the report states that a "more adequate base of 
scientific knowledge fi.e., presumably from fundamental 
re6ea?:cb.— . ad] is required before this applied research 
can pose answerable questions with reasonable tools" 
(p. 65). The implication of this statement is that 
applied research must always be "driven" by fundamental 
research — and such is simply not the case* even in 
such • • esoteric area as high energy physics.*** The 
NAS report also uses Sesame Street as an illustration 
(quite porrectly) of "the ways in which work on an 
• applied problem, can, in turn, advance fundamental in- 
quiry as well" (p. 61). 



However, the NAS report does not pursue this very 
significant point elsewhere in its discussion or in its 
recommendations. Further (as we shall note later), the 
funding recommendations of the NCER Program Committee 
report seem to be oblivious to this point and indeed 



Connolly, T. "Information processing and decision making in organi- 
zations. In B.M. Staw and G.R. Salancik", New Directions in Organi - 
zational Behavior, 1977. Chicago: St. Clair Press. 

*CISST is very familiar with the Connally study because Connally was 
at the time a doctoral student of Dr. Radnor and the research was 
performed under Dr. Radnor's supervision. 

*As seen in the Radnor, Zaltman & Kernaghan (1976) study of this field 
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could have the effect of preventing the advancement 
of fundamental inquiry through more applied contexts* 

* 

Finally, we might note that precisely because the 
\ . distinctions between applied and fundamental research 

in education are not and are not likely to be clear, 
it is not obvious that a change in policy as suggest ad 
by the HAS and NCER 'Program Committee reports would 
have the intended effects, « It is too natural (and 
X easy) for people to "bend" the "descriptions" of 
their work to what is perceived as the "new" policy* 
Thus^ the ^ole issue of Agency/ field relationships 
will be cent^ral to implementation of a change in 
policy, 

7) The NAS report makes a very strongj case that education 
benefits from fundamental research in a variety of 
disciplines. As we will demonstrate later, however, 
there Is a subtle (but potentially very real) danger 
that persons or agencies not already committed to 
education-relevant fundamental research (in contrast 
to the NAS committee members and NCER/NIE) would well 
take the NAS committee's own illustrations and discus- 
sions to conclude that there is no funding role for 
NIE, or evei: that funding for fundamental research 
need not be "earmarked" for education per se con-* 
elusions diametrically opposed to the recommendations 
of the NAS report* 

8) ^^e NAS report does not give consideration to the type 

of management capabilities NIE would need (or what it 
currently has) In order to play an effective role in 
strengthening fundamental research and/or to use increased 
levels of fundamental research funding for this purpose. 
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The .NCER Program Committee report is even more deficient 
here because it does not deal with the management capa- 
* bility issues which would accompany the drastic increases 

in funding levels recommended by this committee. 

• 

9) Finally, as we will discuss later, the NAS committee's 
analysis of statistical funding data (pp. 50-60) is 
inadequate and misleading, the report admits that 

f 

"the data needed to make comparisons are sometimes 
unreliable or unavailable" (p. 50) — but then proceeds 
to use such data and make comparisons as if these data 
weaknesses did not exist. 

Because we believe so strongly that fundamental research relevant 
to education is indeed of critical importance, we have chosen to 
submit this analysis to NCER and NXE. Our intent is not simply 
to be critical of the NAS report. Though we will discuss its 
deficiencies, we have already noted that the report does indeed 
highlight many critical aspects and requirements of fundamental 
research. Rather, our intent is to suggest issues, rationales 
and policy/ strategy alternatives which will supplement and com- 
plement the NAS report and x^hich will provide a more adequate 
basis for consideration of the recommendations of the NCER Program 
Committee. 

We now turn to an examination of some issues critical to an analysis 

* 

of fundamental research relevant to education. 



* 



A more comprehensive discussion of many of these issues is provided 
in an earlier CISST policv analysis for NIE (Radnor, Splvak and 
Hofler 1976). 




II. DEVELOPING A RATIONALE fOR ALLOCATION OF FUNDS TO 
FUNDAMENTAL RESEARCH RELEVANT TO EDUCATION 

The NAS report has reconnended that fundamental research in ' 
education be strengthened a reconsnendatlon with which we 
concur*. It Is clear that such a reconmendatlon has potentially 
significant Implications for siich matters as funding, NIE staffing, 
selection of areas and projects to be funded, etc. It is such 
liqpllcatlons' which the NCER Program Coimaittee addresses in its 
first recommendation ("Allocation of Funds")* Specifically, the 
NCER Program Committee has recommended that: 

At least 20% of the Institute's funds shall be used 
to support fundamental research relevant to education 
by FY 1979, and by FY 1985 this shal^l have Increased 
to at least 307.. 

At least half of the minimum thus allocated (10% of 
the Institute's funds in FY 1979, increasing to 15% 
by 1985) shall be used for research grants to 
individual investigators or small groups of 
investigators. 

While these recommendations attempt to address valid, significant 
Issues, there are some critical problems with these recommendations 
about which NCER and NIE should be aware. The problems include at 
least: the lack of a rationale for the specific target percentages 
an^ dates; the lack of consideration given t:o the educational R&D 
as a total innovation process and to the Impact the funding recom- 
mendations could have on other aspects of i:ducational R&D; the 
lack\of consideration given to critical aspects of the educational 
R&D dontext. As the discussion will indicate, consideration of 
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The NAS report (upon which the NCER Program Committee recoironenda- 
tlons are presumably^ based) is also deficient in considering these 
concern^ * 

-ERIC . ' 'J 
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such Goncerns\ nay well suggest an alternative set of funding 

1. The Need fdr a Rationale for Funding Allocation Decisions - 

First , the Progra^ Coimnlttee provides no specific rationale for 
choosing either the percentage targets or the target years. Pre- 
sumably, the reconmiendatlo^. 4.s^?^^^ V^P^^^'^^^^^ °^ 
the HAS report to strengthen fundamental research relevant to 
education. However, the NAS report neither specifies nor provides 
a rationale for selecting target percentages or years. Thus It 
is not clear why 207. and 30% were chosen instead of, for example, 
157. and 257., or 107. and 357., or 257, and 407., etc. Similarly, It 
Is not clear why 1985 was chosen Instead of, for example, 1980 
(an i^omedlate Increase to a specified percentage) or 1990 (a more 
gradual Increase) . Nor Is a rationale pjrovlded for selecting a 
target year at all. Indeed, we will suggest later that a target 
year not be specified. We are suggesting, then, that while 
decisions about levels of funding are required, such decisions 
must be bused upon sound rationales — rationales not provided by 
the Program Committee. Later in this discussion* we will siiggesc 
such rationales. 

Second , while It is valid to consider the need for strengthening 
fundamental research relevant to education, this cannot be done 
in isolation from consideration of other areas of educatlonel 
R&D and of NIE's responsibilities In tht se areas. Specifically, 
NIE also has responsibility for awd must give consideration to 



it 

In its discussion (p. 54) of Table 3 (p. 55), the NAS report does 
mention a "one-third" figure. However, this referred to funda- 
ment<il research as being one- third of total research allocations, 
not one- third of a total budget. 
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the educational R&D-related functions of need Identification, applied 
research, development and dissemination, and implementation/ utillzatlcn* 
In light of its responsibility for educational pro*-lems and practice, 
NIE must also give consideration to such matters '^Ticy research 
and training/technical assistance for users of • .ional R&D outputs. 
In addition to considering each of these areas individually , NIE must 
also give consideration ro orchestration and system building across 
these areas. Simply put, NIE msut consider educational RfJ} as a total 
process of innovation. Thus, there are a number of critical concerns 
which must be considered in development of policies and strategies 
for strengthening fundamental research. However, these are concerns 
which are not addressed by the MCER Program Committee in its recommen^ 
datlon and only marginally by the NAS report. 

t 

A third area of concern which must be consldeired Is the actual 

context of fundamental research In education* Critical questions 

must be raised as to the rate nt which quality fundamental research 

relevant to education can be built up; the ways In which the multl- 

dlsclpllnaryy derivative nature of education might Impact policies 

and strategies for strengthening educatlon^relevant fundamental 

research; the fact that funding for education-relevant fundam^tal 

research In a variety of disciplines Is provided by a variety of 

federal agencies; etc. Again, these are critical Issues not 

addressed by the NCER Program Committee recommendations or by the 

NAS report. As we have noted elsewhere, consideration of sUch 

Issues leads to consideration of both funding and non-«fundlng 

if 

policies and strategies* Here» however, we will limit the 
discussion here to the Issue of funding allocations, 

\ 

^Radnor, Splvak and Hof ler (1976) . Increasing funding Is perhaps the 
most *'obvlous'' strategy for strengthening fundamental research relevant 
to education. It Is very Important to recognize, however, that ' 
strengthening fundainental research In the education context Is not 
just a funding Issue* It also Involves developing linkages, building 
system capacity, selecting projects and areas for research, etc. 
These are Issues we will be discussing later In this paper. 

\ 



11- 



2# Building a Rationale for Fuudli ^ ^ Allocations 

There are at least several critical considerations upon vhich 
funding decisions for education- relevant fundamental research 
should be based* (We are assuming here, a common agreement 
that education-relevant fundamental research is indeed important 0 

First s excellence is critical to fundamental research in any field* 
As the NAS report correctly notes (p* 5), ^'relevance** in fundamental 
research is essentially meaningless if the research is of low 
quality* Funding low Quality fundamental research programs or 
projects may add quantity but not quality to the fundamental 
research base and would simply be dysfunctional and wasteful* 
Further, we ma^ note that funding levels which are greater than 
can be used effectively by high quality researchers /research 
institutions invites application by and distribution of funds to 
low quality researchers /research institutions* Thus,, as the NAS 
report emphasis s, funding must be provided in a manner which 
carefully selects high quality programs/projects and ''screens 
out'' low quality programs/projects* 

Second , if excellence is a critical requirement, then the capability 
of the field to absorb and use productively Increased levels of 
funding for fundamental research programs /projects is limited by the 
available base of high quality fundamental researchers and funda- 
mental research institutions • This Is a funding implication which 
is not raised as an issue by the NAS report* Simply put: the fewer 
the number of available high quality fundamental research personnel, 
the lower will be the level of funding for fundamental research projects 
which can be productive. 

Third, the current base of fundamental researchers and research 

institutions also limits the rate at which the fundamental research 

\ 



Current and as augmentable in the near term* 
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field can be "built up" — for it is the existing fundamental 
research personnel and institutions which provide training for 
new fundamental research personnel. Further, training of new, 
high quality fundamental reseatchers does take time. These con- 
siderations are constraints on the ujillty to "in^rove" or "strengthen' 
fundamental research through increased levels of funding — and 
again, these are issues not considered In the NAS report.j 

There ||p a basic question concerning the structure of fundamental 
research relevant to education that must be dealt with and which 
was not considered in the NAS leport. Granted that education 
has historically been a field that is practice-based and largely 
derivative of research from a variety of disciplines, the 
question arises as to how important It la, if at all, that there 
be a core of fundamental research activity within education Itself 
(and/^r by researchers whose\ long-term and major commitment is to 
education). Is it likely to be healthy for education to be 
entirely a derivative field ar is there a need for a significant 
level of research effort in Ifundamental areas that comes from 
within education (possibly in some loosely defined sense) 
It is 9lear that the core of researchers that are primarily 
committed to education has been small and weak to date though 
it does exist (and was even represented on the NAS committee 
making the study). A major policy issue is the extent to and 
rate at which such a core should and can be built. Furthermore, it Is 
vital that we better understand the extent to which the size 
and health of such a core does or does not represent a constraint 
on the healthy growth an4^ flow of useful insights that can come 
from supporting potentially relevant research in the contributory 

^ ( 

■ 

* We are definitely not x^uplying by this the need for such work to go 
on only in schools of education. The issue is one of coramitment. 
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disciplines. And finally, what is to be NIE's role with respect 
to fundamental research carried out by those with major versus 
peripheral commitments to education, as compared to other sources 
of funding tor -these areas of fundamental research? Such questiotis 
are not easily resolved. They should have been an important 
aspect of the NAS analysis. They should be the subject of further ^ 
study by NIE. We take a tentative position that such a core is 
likely to be important and that education has suffered from the 
lacVc of such a strong core. 

In the present educational c^ontext, the existing base of high quality 
educational fundamental resejarchers (i.e., those committed strongly 
to education) is small. Thip would imply some constraint on ftinding 
levels. On the other hand (as has been well noted by the NAS 
report) , education can derive benefits f i^om fundamental research 
in other disciplines which do have established, sizeable, high 
quality personnel/institutional bases. Here, however, thbre is 
also a constraint. Education research tends to have iCw prestiee, 
complicating efforts to attract qompetent researchers to this 
field. 

A fourth consideration is that, educational R&D includes not only 
fundamental research but also need identification applied research, 
development, dissemination and other areas relevant to educational 
R&D and its application within the operational system of education. 
Here, consideration m'ist be given to the need to balance NIE's 
responsibilities across each of these areas; to the need for a 
balanced rate and level of development across each of these arcaj;; 
and to NIE's responsibility for the impact of educational R&D on 
the educational operating system. Here, then, several basfc 
questions must be asked which are not asked in either the NAS or 
NCER Program Committee reports. 
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I) What are the needd in all of these areas for which NIE 
has responsibility? 

ft 

The Academy report has concluded that education- 
's relevant fundamental research should be strengthened. 

Howevar , because educational R&D is relatively young 
in this country, all educational R&D areas need 
strengthening, though in varying degrees. In particular, 
we would note that some people consider need identification, 
applied research, disseminatlonand Inplenentation/ultili- 
' zation to be areas which have even more "need" for 

"strengthening" in the educational context than does 
\ fundamertal research. 

2) ' What are the relative levels of funding requiri d for 
significant Impact across the various educational R£dD 
related areas? 

Thfe point to be made here is simply that compared to 
applied research and development, the costs for slgnl** 
f leant fundamental research are relatively small. 
Both app Heches earch and development have requirements 
of scale several degrees of^ magnitude significantly 
larger than the scale requir<i^ments of fundamental 
' research. Differences in sectors and disciplines make 
precise analogies difficult.* In areas su^ as drugs and 
chemicals (which involve "hard" sciences and which have 
very high levels of industry involvement), funding may 
be as much as ten times greater for applied research 
(and one hundred times greater for development) than 
for fundamental research. In the "softer" social 
sciences, the differences in magnitudes of funding 
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We recognize that in the education context, applied research Is 
often me in a "small-scale" mode. However, the fact that applied 
research on a larje-scale mode appears to be largely absent in the 
education context may be a weakness worthy of ^concern. 



may not be sV) great — *but they are still significant. 
In the social sciences, fundj.ng for applied research 
trptcally is two-to-fout times greater than for 
fundamental research, with the ratio between develop-- 
ment and fvindamental research being even higher. 

The* in^llcatlon of the above is that funding for funda- 
mental research can be Increased significantly without 
undermining the funds available (and needed) for other 
R&D functions. At the same tlme^ there Is also an 
Inqpllcat^on that^ funding levels for fundamental 
research are not typically 207^ to 30% of the total 
combined funding for fundamental research, applied 
research and development • 



What impact would various levels^of funding for funda^ 
mental research have on NTK's ability jto fulfill its 
responsibilities in oti educational R&D areas? 



The NCER Program Committee lias recommended that 20% 
of NIE*s budget be allocated to fundamental research 
by 1979 and 307» by 1983. This recommendation can be 
properly evaluated only in the context of NIE's 
responsibilities for all aspects of educational R&D 
(need identif Icatlon, fundamental research, applied 
research and development) , as well as fof dissemina- 
tion and t^lementation/utlllzation of the outcomes 
of educational R&D. Allocation of 30% of NIE's budget 
to fundamental research 4/ould leave only 70% of NIE's , 
budget for its other R&D -related responsibilities. 
Since applied research and development both tend to ^ 
require much higher magnitudes of funding than does 
fundamental research, the effect of the NCER Program 
Comrt^ttee's recommendation Would likely be to weaken NIE 
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capacitry-to meet funding needs for applied research and 
development -- as well as funding needs for l^IE's other 
areas of respotislbllity 

Consideration must also be given to the fact that NIE 
does,, have fixed expenses for administration (12-14% of 
Its budget In recent years) and other commitments for 
which funding Is allocated (e.g.: contractual coranlt- 
ments; programmatic commitments such as the Implied 
three to five year commitments to SEAs Involved In 
the State Capacity Building Program; Congress lonally 
mandated commitments such, as are found in NIE*s rer* 
authorizing legislation).* Fixed expenses knd commitments 
cannot be automatically removed or reduced to re-allocate 
funds to fundamental research (or to any other H&D-*related 
area of concern). Since fixed expense*8 and commitments 
already represent a significant portion of NIE*s budget/ 
increasing NIE's fundamental research allocation to 20-307. 
could be acconq)llshed only by using funds from the 
"uncommitted" portion of NIE's budget — thereby 
significantly reducing the level of the discretionary 
portion of NIE's budget. This single fact raises at 
least two serious questions about the NCER Program 
Committee ' s recommendations . 

a) Would th^ "remaining** uncommitted portion 
of NIE's budget be adequate for NIE to 
fulfill Its other R&D-related responsibilities? 

b) Would NIE have the level of flexibility and 
discretionary use of funds it needs to respond 
to anticipated or unanticipated changes in the 
educational R&D context, promising new development 
in any area of the total educational R&D , 
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process, the need. to "balance" the levels 
•<; and rates of maturation across all educa- 

tional R&D fui^ctions, etc.? 

It must be noted that the above discussion is not meant to imply 
or suggest specific funding amounts or percentages for fundamental 
research or for any aspect of educational RSiD. Such determinations 
arc the responsibility of NIE and NCER and are beyond our capability 
or intent. What is intended is to suggest that the issues discussed 
above are the types of considerations which should be used to provide 
the rationale for decisions about specific funding allocations for 
fundamental research and indeed for all funding allocations* 

« 

It Is also intended here to suggest that allocating 20 - 30% of NIE*8 
bt^dget to fundamental research is quite likely to be dysfunctional to 
NIE* 8 capability to meet its total responsibilities.. 

There is one further comment we would make here. Stability over 
time is very important for fundamental research. Thus, while we 
recognize that percentages can be a useful tool for administrators, 
we do question the wisdom of allocating a total budget percentage 
(10%, 20%, 30% or whatever) for fundamental research. If NIE's 
budget were to increase significantly, a total budget percentage basis 
might well result in more funding being a8sip,ned than the personnel/ 
institutional base of fundamental research could usefully absorb. 
Contrarily, if at sometime NIE's budget were to decrease significantly, 
total budget percentage basis would require cutting funding alloca- 
tions to fundamental research projects — thereby introducing an 
instability which would likely be dysfunctional. 

3, An Alternative Funding Strategy for Fundamental Research 

iVTtght of the above discussion, the need is for a funding 
policy/sbcategy for strengthening fundamental research which is both 
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effective and feasible (given constraints discussed above and the 
need for "balance") . Based upon the above discussion, we would 
suggest the following as an t.ffecjtlve and feasible alternative. 
While this alternative assumes there will likely be an Increase 
In the level of funding for ■ fundamental research' relevant to 
education, we do not here Intend 4:0 suggest a specific level of 
funding. That would be beyond our capabilities and Is the 
responsibility of NIE. This alternative also assumes that analyses 
and estimates will be made (and taken Into consideration) about 
such Issues as: th^ levels of funding which the fundamental 
research personnel/Institutional base Is currently capable of 
using productively; che relative need for funding across all 
aspects of the educational R&D process; etc. 

1) NIE would establish a target-dollar'amount level for 
NIE funding of fundamental research relevant to educa- 
tion. The target dollar amount level would. likely be . 
greater than the current dollar amount level. 

\ 

Discussion : The use of a target-dollar- amount level for 
Increasing NIE funding of fundamental research would 
serve several purposes and would have several advantages 
over a percentage-of-budget basis for funding. Both dollar- 
amount and percentage-of-budget formats would demonstrate 
the cotonltment of NIE to a policy of strengthening funda- 
mental research/ Both would provide goals or targets** to 
be reached* However, a target* dollar-amount basis for 
funding would also allow NIE to relate funding to the 
capability of the field to use funding productively and would 
protect fundamental research funding from f luctua^t/lon or In- 
. stability In NIE*s budget* A percentage-of-budget basis for 
^ funding would not serve these purposes* 

i 



The specific target-dollar •amount level would be established 
on th^ basis of realistic estimates of probable NXE budget 
levels, the capability of the field to productivf&ly use 
fui\fiing and consideration of NIE's mission responsibilities. 

V 

2) The target-4ollar«» amount level would be reviewed at approxi- 
mately five, year /intervals. 

Discussion ! It is not assumed that a specific target-dollar- 
amount level, once set, is either "written in stone for all 
times*' or based on "perfect wisdom". The target level doea 
- represent (1) a definite commitment which (2) la" based 
on well-considered judgment. At the same time, there 
must be review mechanisms which can take Into account 
auch context changes as significant changes In NIE's 
overall budget levels. Increased capability of the 
field to use funding productively, etc. 

While a review process Is needed, t^e process must be 
neither too often nor too seldom. ,If performed too often 
(e.g.: annually), there Is not adequate time to "measure" 
and ''weigh" the changes noted aboye; the danger of Insta- 
bility is re- introduced; and the concept and force of 
"commitment" is undermined. Contrarlly, if the review is 
performed too seldom, (e.g.: every 10 years or more), 
adaptation to context changes may come "too late" to be 
effective and the concept of "review" becomes operationally 
meaningless. Tfivijs, we suggest the review be performed 
approximately every five years. 



*0n the assumptions that NIE's overall budget level is a key factor 
and that it is roost subject to significant change as a result of 
political dynamics, consideration might be given to a four- year 
review cycle, performed in the second year of each Presidential 
term. There would be both benefits and dangers to this approach, 
but discussion of these Is beyond our scope here. 
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3) In order to reach the target-dollar-amount level specified 
as policy, NIE would establish as policy an annual-lncre- 
mental'percentage growth of NIE's fundamental research bi.idget< 

„ Discussion ; Any increase in NIE funding for fundamental 
research must take into account (1) the current capability 
of the field to use productively increased levels of 
funding and (2) the rate of development of *he field over 
time. An annual-incremental-percentage basis ^r 
increasing fundamental research funding takes both of 
these factors into account. Indeed, the specific 
annual percentage set as rolicy would l^e based on 
informed estimates of these two 'factors. Additionally, 
this strategy is based on the perspective that the 
nature of fundamental research in general (and 
specifically in the education context) calls for a 
steady rather than a one time/short term "spectacular" 
rate of growth. The specific annual percentage would 
also be subject to review approximately every five years 
for the same reasons as noted above* 

The percentage formula recommended here is a percentage 
of NIE's fundamental research budget , not of NIE's 
total budget. 

4) NIE would establish as policy that funds set aside for 
fundamental research would be used only when programs 
or projects meet the crite ion'^of researcher and/or 
ref'earch institution excellence (and any other criteria 
established by NCER/NIE) . 



If deemed wise and feasible, the incremental percentage coxild be 
higher in the first year both in order to "signal" ME's commitment 
to the field and to bring the level of fundamental research up to 
the level at which the field is currently able to use funding 
productively. 
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Dlscuaslon : This . policy operationallzes the valid concern 
of the MAS committee that low quality fundamental research 
is meaningless:' no matter how '^relevant^^ It might appear. 

5) Provision yould be made that for any given fiscal year, 

* the annual percentage growth may be temporarily suspended 

for a one-y^ar period » with the complementary provision that 

the level of| funding already reached would not be reduced^ 

*■. I ■ ' 

/ 

Dlsr.uaalon : We have stressed the Importance of. not providing 
levels of fundamental research beyond the capability of th^\ 
field to use productively. An Inflexible growth In funding \ 
levels does not take this factor Into account* The above 
provision provides the needed flexibility . — I.e. » If at 
any time NIE finds that the level of productive funding has 
"peaked" (temporarily) , It may suspend the annual Increase 
In funding levels* Simultaneously, stability would be 
provided In that the level of funding already leached would 
not be reduced. The Intent of a policy of annual Increases 
In funding level would be protected by the provision that a 
suspension would be for one year only* Continuation of the 
suspension (if valid) would require an annual decision* 

6) No target year would be established for reaching the 
target ^liar-amount level. 

Discussion : Setting a target year my "sound good" at 

- -' - \ 

first glance, hut it is unnecessary ^nd the effect is simply 

to rigldlfy the process, thereby leaving NIE less able t;o 

» 

adjust the process to the capability of the field or to 
changing conditioria (e.g.: a period in which the Congress 
does not provide for the increases in NIE's budget needed 
to increase allocations to fundamental researcla). 

'5: 
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As part of the liiq>lementatlon of the above policies, 
NIE would continually monitor tl^| capability o,f the 
field to use fundamental research funding productively. 

Discussion ; This p revision} necessai^y for the 
implementation of the policies and strategies suggested ^ 
above, could well be Implemented as part of a total KPU 
monitoring process. This provision would also serve the 
"secondary" purpose of * developing/maintaining a close 
contact between NIE and the fiel4. 



-23- 



III. STRENGTHENING FUNDA^IENTAL RESEARCH; STRATEGIES AND ISSUES 

.. : ' CT"^ 

The basic conclusions o£' the NAS ];«port are that fundamental research 
V relevant to education is important, that it has been given low 

priority and that it should be given higher priority. In a word,, 
the basic conclusion of the NAS report is that fundamental research 
'relevant to 'education should be strengthened. Given this premise, 
it becomes critical to determine how this* "strengthening" can and , 

should be accomplished He, , how the objective of "strengthening" 

can and should be operationalized* 

. . •• • ^ 

Probably the most obvious approach is to increase the level of funding 
allocated for fundamental research relevant to education. However, as \ 
the discussion above has implied. Increased funding must be. seen as only 
one aspect (admittedly an important aspect) of an overall set of inter- 
active policies and strategies. Thus, it now becomes critical to ask 
the "larger picture" questions: What is th^ahge of policies and 
strategies which might be considered? What ar^ their "pros" and 
"cons"? How do they fit into a synergistic "portfolio" of policies 

• . hpA strategies? What are the critical set of issues involved? What can 

" and should be the relevant roles of NIE and the field? 



\ 



Both the NAS and the NCER Program Committee reports recommend a set of 
policiefe an(i strategies which do Indeed go beyond the "level of funding" 
issue. Nonetheless, there are significant strategies and issues which 
are not (or not adequately) considered ir. either report. 



The concept and importatce of a 'Jportfolio" approach is discussed in 

6' ' 

r 



Radnor, Spivak and Hofler (19X6)^. 
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I. Expanding the Purposes of Fundamental Research Fut^ing 

/It is reasonable to assume that a major purpose o£ increasing the 
level of fundamental research funding is to increase (through an 
increase in the number of projects funded) the bavSe of scientific 
knowledge about education* This is a very valid purpose* At the 
same tlmei there are other purposes which are suggested by the \s 
nature and context of fundamental research relevant to education! 
Specifically, as we have noted in an earlier analysis for NXE; ^ 



The central core of Basic Researchers, those comnitted 
^to and devoting the bulk of their careers to educational 
Research, seems to be particularly weak in this field » 
especially when compared to Basic Research in mpst other 
fields. In education, this cere group tends for the most 
part to be located in schools ot^ education, in depart- 
ments focused on each of a number of derivative disciplines 
(e.g., educational psychology, educational sociology). 
'Consideration of these settings as the primaryX institutional 
bases for Basic Research in education suggests ^hat there 
are relatively few centers of excellence and a gi^at deal 
of mediocrity. Also, when one makes the distinctlo-a 
between numbers staffing these departruents and numbers 
carrying out significant amounts of Basic Research, \the 
relatively small size of this core, and its scattered 
condition, become apparent* \ 

A somewhat different picture emerges when one examines 
Basic ReseaY*'jh relevant to education ca?:ried out in 
discipline-based university departme^s (e«g«, departments 
of Psychology or Sociology). Here, ^here is considerable^ 
Excellence, and these disciplines often provide valuable ; 
inputs and contributions to the education knowledge and |\ 
technology base* The problem^ however, is that tb^ 
commitment of discipline^^based Researchers to the field 
of education tends to be variable and shifting DVer time* 
After all, their primary coimnitment is to their discipline, 
and education takes a secondary role* 

Taken together, these two conditions create an educa- 
tions'. Research community that tends to be unstable and 
amorpnoua, thus complicating the problem of relationships 
and of inalntaining communication flows among the parts <^ 
the system* (Radnor, Spivak and Hofler 1976, pp* 29-30) 



ic 

As noted earlier, we recognize that the need for a "core" of 
educatlon-orlented fundamental researchers is an issue which 
needs further discussion* 



The In^llcatlon of the above discussion is that fundamental research 
project funding should be based on a multiple purpose strategy. 
That is, in addition to the purpose of strengthening the base of 
scientific knowledge about education, project funding decisions 
should take into consideration such additional? system building 
pjurg^ses as: 

. • 

1) creating a vital, stable core of high qxiality ' 
'fundamental researchers who are committed' to 
education-oriented res«arch ; * 

2) facilitating and sustaining communication and 
collaboration across education- relevant 
disciplines and across the many agencies and 
Institutions which fund fundamental research 
relevant to education* 

These purposes Imply project funding strategies which seek to maintain 
relationships with researchers beyond the life of a specific project, 
which seek to develop "centers of excellence", which support training 
programs for educational fundamental researchers. Indeed, these 
purposes and strategies would appear to be essential if NIE is to 
take seriously its mandate to "build an effective educational R&D 
system. " While Recommendation No. 7 of the NAS report (and related 
discussion on p. 75) begins to address such purposes and strategies, 
we believe that such system building purposes and strategies of 
fundamental research funding should be given considerably more emphasis 
and vconsideration than provided by the report. 

There is one further issue that should be considered here — the 
politi al issue. NIE is a federal agency and as such it is (and has 
been) -ubject to "political" dynamics. Because "political" dynamics 
tend to be "short term" in nature, they pose a potential threat to 
the more long term stability needed for fundamental research. With 
this point in mind, we are concerned that by focusing attention on the 
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Importance o£ fundamental research ''outputs", the NAS report might 
lead to Congressional expectations of short term "results" — despite 
the valid 'cautions of the report to the contrary. In making its 
"case" to the Congress, It would seem advisable for NIE to focus 
(1) on the need for strengthening the educational fundamental 
research community (and its linkages) and (2) on multipurpose 
strategies for doing so with the Illustrations of the NAS 
report providing support for the long-term validity of system-building. 

2. Selection Decisions; Projects and Areas of Fundamental Research 

NIE specifically requested that the NAS committee Identify areas of 
fundamental research having potential significance for education \nd 
describe areas of research they would deem to deserve high priority. 
Clearly, if NIE is to be Involved in the funding of fundamental 
research rej-evant to education, NIE staff must be provided with some 
guidance from the field to inform the inevitable selection decisions 
that will have to be made. Whether the fundamental research funding 
policies of NIE entail (1) a focus on a few areas in which projects 
will be funded or (2) a totally open, 1007. field initiated approach, 
choices will have to be made from among proposals submitted. Further, 
since a variety of disciplines are relevant to education, there is 
potentially a broad array of relevant research areas from which 
project proposals could be submitted. 

Thus, some criteria other than the relative merits of submitted 
proposals are likely to (and indeed must) enter into selection 
decisions — even when panels of distinguished researchers are 
used to make selection decisions* Distinguished researchers, 
after all, do have their own individual biases and perspectives. 
Thus, the research areas, methodologi-es^ etct that are funded are 
likely to be heavily influenced by the simple act of selecting 
distinguished researchers for the review panel. Therefore, there 
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is really no way to avoid the problem; decisions are going to 
rQ£lect the conscious or unconscious choices among research areas 
as having greater or lesser potential relevance for education, 
potential significance for strengthening the scientific foundations 
of education, etc. 

It Is reasonable to suggest that NIE's fundamentel research funding 
decisions need to be based on some understanding of at least the 
following: 

1) the spectrum of fundamental research areas with potential 
relevance to education; 

2) the state of development of each of these ' research areas; 

3) lines of inquiry that are being pursued; how, where 
and by whom; with what kind of results, and how such 
results might have potentially significant application 
to education; 

4) the state of readiness for educational application in 
different areas of inquiry (i.e., are there some lines 
of inquiry that are so well developed that it becomes 
feasible to concentrate on a few central problems of 
questions which may lead to insights and/or outcomes 

of fundamental significance to improving our understanding 
of the educational process or how we think about 
education?) ; 

5) how well different areas are (or are not) being funded 
by other federal agencies; 

6) which areas seem to have reached a temporary saturatidn 
for funding high quality fundamental research; 

7) which areas could expand most rapidly with the help 

of increased funding (given the availability of personnel and 
centers of excellence able to expand their training 
capacity at different rates); 

8) where the potential for synergy across agencies 
and/or projects seems most likely. 



\ 

/ 
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We do not here deny the argument made by the NAS committee ..^hat 
given the uncertain nature of fundamental research, one cait never 
predict with complete accuracy where the most Important break- tl^roughs 
are likely to be, what applications are likely to follow from .what' 

m 

lines of fundamental research, etc. Still, some reasonable. 
Informed estlmatns ere not outside the realm of possibility, and 

this seems to be ail that.MXE was requesting from the NAS cooanittee^ 
Funding decisions will be made regardless of what recommendations 
the NAS committee makes; and slnc^ decisions are almost always made 
on less than perfect information, any information which could 
^-reduce the uncertainty level of such decisions by improving the 

quality of information available to decision makers would certainly 
seem to be a decided plus» Distinguished researchers from the 
field would seem to be the best possible source for at least part 
of this information. Thus, it is a serious falling of the NAS 
committee that they were unwilling to provide any of the guidance 
^'^^ requested abc w "promising^' areas for fundamental research. 
In effect, the NAS committee took the far easier task of selecting 
illustrative examples aimed not at providing guidance for selection 
decisions but rather simply at persuading the reader that fundamental 
research does have potential relevance for education. Certainly NIE 
and NC£R do not need to be so persuaded. By redefining their task 
and thus producing the kind of report they have presented, the NAS 
committee has, in our view, provided a potentially useful public 
relations documeQit (though as we will later note, it could have a 
negative effect on some readers) in support of fundamental research 
relevant to education, but has provided little of the kind of 
guidance NIE needs to assist its staff in developing a fundamental 
research funding policy or making specific decisions regarding the 
implementation of such policies. 

3. The Role of NIE 
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By virtue of its Congressional mandate to ''build an effective 
educational R&D system", it is obvious that NIE does have a role 
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In the strengthening of fundamental research relevanfr'^to education. 
It is ^hus now time to examine the nature of that role. To do so, 

. it' is necessary to consider the context in. which NIE exists and in 
which fuiidamental research relevant to education occurs — a context 
which includes^ the multi-disciplinary, derivative nature of 
fundamental research relevant to education; a relatively young 
edutrational fundamental research system in which there are many 
loosely-linked participants;/ several ma^or federal (and other) 

' funding agencies (of- which NXE is only one) . 

The educational context for fundamental research has two main 

"characteristics. First, it is multi-faceted. Educational 

fundamei^al reseai;ch is derivative from many disciplines and 

thus Involves many participants, each f/fth somewhat different 

perspectives and interests. There are several major federal 

(and non-federal) finding agencies involved — of which, NIE 

is only one. Seco'^nd^'^ the educational fundamental research 
community is relatively young (rather than "mature") — and its 

li-nkages are we^k. In such a context, there are three particular 
"lead" roles which seem necessary but which could only be provided 
by an agency or institution whos^ scope is nationwide and multi- 
disciplinary and whose prJtmdKy mission is education-oriented. NIE 
is Just such an institution. Indeed, given its Congressional 
mandate, NIE is the only institution in a position to perform 
these very necessary roles. 

As has already been implitid In the discussion thus far, one NIE 
role is to provide leadership in system-building . At the very 
least, this role implies the need for policies and strategies 
which would: 



(1) facilitate the development of linkages among disciplines, 
among funding agencies, and among fundamental research 
personnel; 

(2) provide for the training of education-oriented fu^idamental 
researchers ; 
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(3) build and strengthen "centers of excellence^ for education- 
relevant ftmdamental research; 

(4) provide stability; 

(5) increase the "status" of educational research. 

A system building role also implieis that the system-building 
potential of a specific project should be a major -criterion for 
project selection decisions* 

A second role of NIE is closely related; i.e., the role of orchestratl 
At the very least, this role implies: 

(1) facilitating communication across disciplines and among 
fundamental research personnel; 
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(2) taking a lead role to identify particular areas of 
fundamental research which appear to have particular 
significance for education; 

(3) taking a lead role to coordinate fundamental research 
programs and projects across funding agencies in order 
to develop synergy across such programs and projects; 

(4) linking fundamental research to "upstream" and "downstream" 
R6tD functions. 

Implied In the orchestration role is yet a third -role for NIE — a 
coalescing role. We may note that fundamental research relevant to 
education may be characterized as 'being highly "scattered". It 
Involves a variety of disciplines — each with a variety of research 
areas potentially relevant to education. Funding for education-rele- 
vant fundamental research is "scattered" across many funding agencies. 
Education is generally a "secondary" concern in these varied 
disciplines and funding agencies. The education-relevant fundamental^ 
research "conmunity" Is at best loosely and weakly linked. Educa- 
tion-relevant research is not per se of "high status". Thus, there . 

4, . 
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would seem to be a strong need for coalescina the diffuse, scattered 
^elements of education -* relevant fundamental research* At the very 
least* this role would Imply: 

(1) scanning the various education-relevant disciplines to 
Identify potential for synergy across lines of research; 

(2) facilitating and supportljog the development of linkages 
within and across fields. ^ 

J 

A. The Agency/Field Issue 

A basic issue in R&D is the role of the field in relation to the 
role of a major funding agency, an issue which we have addressed 
in a policy analysis for NIE (Radnor, Splvak and Hofler 1976). 
Both the NAS and- the NCER Program Committee reports focus strongly 
on this issue in their recommendations. Though their recommendations 
differ somewhat in specific details , both reports recommend the 
fallowing kinds of roles for the field: 

1. A review role (e.g. : to review proposals » quality and 
balance of programs, etc.); 

. 2. A role of assistance to NIE (e«g«: to identify research 
needs, to develop guidelines, to plan); 

3« To Initiate fundamental research projects. 

Additionally, the NCER Program Coimiittee recommends that 50% of 
NIE's fundamental research funding be allocated to ••individual 
Investigators or small groups of investigators." 

Both reports, then, suggest a strong role for the field. However, 
there are issues which we feel need further consideration and clari- 
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flcatlon than provided in either report. While we have discussed 
these issues at more length elsewhere (Radnor, Spivak and Hofler 
1976) , we will review them briefly here. 

One issue is the level of consensus among researchers in the field 
about such considerations as: identification of key questions in need 
of answers, adequacy or appropriateness of different methodologies, 
lines of research which appear to be most promising, etc. Where 
field consensus is low, it would seem more appropriate for a 
funding agency to work closely with (but clearly not be directed 
by) the field to determine areas and projects for funding. That 
is, the agency would "be fairly active in molding and selecting 
from what the field has to offer rather than just responsing to 

it 

scattered field- initiated proposals" . On the other hand, where 
field consensus on key issues is high, such an Agency role would 
seem less necessary — ie. , a mode of being highly responsive to 
field initiated proposals would be quite appropriate. Thus, it 

« 

is incumbent on NIE to determine the level of field- consensus on 
key fundamental research issues before setting a fixed /policy about 
field initiated proposals. However, pending such a determination, 
it appears to us that in many areas, fundamental research relevant 
to education is more nearly characterized by low rather than. high 
consensus. If this is indeed the case, the "lead" roles of 
orchestration and coalescing become especially important. 

A second (and related) Issue Is the need for system capacity 
building In relation to fundamental research relevant to education. 
As we noted earlier, this need Implies that projects should be 
considered In terms of their potential for system capacity building 
as well as In terms of their quality and substantive output. An 
"overview" agency such as NIE is more likely than the field to 
have an "overview" perspective of total system building needs and 
to be specifically concerned with this Issue. 
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The basic Implication of the above discussion is simply that there 
are "lead" roles which can only be performed by a major funding 
agency such as NIE and that these "lead agency" roles are especially 
Important when an R&D system such as that in education is relatively 
young and not really "mature". We do note here that while NIE may 
validly be considered a lead agency for fundamental research 
relevant to education, there are other federal agencies (e.g., NSF) 
which fund fundamental research relevant to education — some 
at levels greater than that provided by NIE. Thus, NIE's "lead" 
role must be understood In terms of the Intersection and inter- 
action of its lead role with the lead roles of other relevant 
funding agencies. 

Having said this, it is now important to balance the discussion 
by again noting that both the NAS and NCER Program Committee 
reports are correct In emphasizing that the field must have a 
strong role In the funding process for fundamental research 
relevant to education. Fundamental research Is an uncertain 
process. It is difficult to determine which lines of research 
are "most promising." The knowledge and perspectives of researchers 
are needed if adequate evaluations are to be made of the quality 
of research proposals. 

The practical question at this point might seem to be: How can 
the "lead" role of NIE and the "strong" role of the field be 
reconciled? Actually, this is the wrong question — it implies 
that the Agency/field ■«ssue is an "either /or" issue. We would 
suggest instead that the nature of fundamental research in the 
educational context requires a highly collaborative Agency/field 
relationship. To illustrate, we have suggested that NIE should 
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not be directed bjj. the field in project selection. However, 
the field should be strongly involved in reviewing project 
proposals — as is validly recommended by both the NAS and NCER 
Program Connnittee reports. Sijoilarly,. while NIE must take a 
"lead" role in the process of identifying areas of "promising" 
fi^amental research relevant to .education, it must seek the 
advice and counsel of the field. 

In order to develop and maintain Such a close, collaborative 
relationship with the field, we would suggest (as does the NCER 
Program Committee report) that NIE have fundamental researchers 
on its staff* — though we recognize that attracting quality research 
talent to NIE is not a simple task. Indeed, we would recommend 
that NIE research personnel continue to be involved in research 
and have as a major responsibility maintaining close contacts with 
other members of the field* 

Through a close, collaborative mode of Agency/field relationships, 
the Agency/field "issue" is no longer an "either/or", "us vs. 
them" issue. Rather it becomes simply an issue of determining 
the appropriate modes of collaboration. Indeed, if NIE has 
active researchers as part of its internal staff , NIE becomes , 
in one sense, a part of the field. 

5, A Potentially Dangerous Alternative Conclusion 

The NAS report was prepared by a committee composed of personnel 
who candidly admit their strong and pre-established commitment to 



*The NAS report recommends only that NIE staff be "well informed 
about research" (p. 79). 
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fundamental research. It is not surprising, then, that the NAS 
committee concluded that fundamental research Is Indeed Important 
for education,. SlmlXarly, the report was requested by and^sub- 
mltted to NIE » which has Its mission the support of educational 
R&D. Thus, It would be reasonable to expect NIE (and NCER) to be 
supportive of this conclusion. 

The Issue here, then. Is not whether the conclusion of the NAS 
report Is right or wrong — * or whether groups such as those above 
will find the report persuasive. Rather, we are concerned about 
the Impact the HAS report might have on persons or groups who are 
not already '^predisposed'' towards (or who may even be predisposed 
against) the conclusions of the report. Given the political 
context in which NIE (and Itr funding) exist, this is just as 
important an issue as the validity of the report's conclusion. 
We think that the report could, unfortunately, be misinterpreted 

it 

by persons "un-predisposed" towards fundamental research. 

In order to demonstrate forcefully the need for educational funda- 
mental research, much of the NAS report is devoted to demonstrating 

that fundamental research in a variety: of disclr lines has provided 

(and thus can be anticipated In the future to provl4e) benefits 

for education — and this argimient of the report Is Indeed persuasive. 

However, It is precisely this persuasiveness which could provide 

a basis for what we consider to be wrong and dysfunctional conclusions. 

Specifically^ there are two possible Issues here: the allocation 

of fundamental research funding with a specific focus on education; 

and the role of NIE. 



Subsequent to the initial draft of this policy analysis, at least 
one Congressional staff member has raised the type of Issues discussed 
in the paragraphs which follow* 
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The NAS report has correctly noted that It is difficult to predict 
which specific lines of fundamental research (and in which disciplines) 
are likely to be most significant for education. At the same time, 
the report does make a sti^ong case that education has indeed benefited 
from fundamental research in a variety of disciplines. If these 
two (quite valid) considerations are taken alone and out of the 
larger educational R&D context (as an "un-predisposed" reader might 
well do), it would not be unreasonable to suggest that funding 
allocations for fundamental re .earch need have no specific reference 
to education per se. Such a position does have some validity — 
i.e., education Aan eJfpect to benefit from fundamental research 
in a variety of fields regardless of Whether or not the term 
"education" is applied. The NAS committee does not provide a 
rationale to the contrary, and indeed, does not really raise or 
discuss th^Tissue. We believe the issue should be raised and a 
rationale delineated. Specifically we suggest (as discussed 
earlier) that it is possible to make reasonable, informed judg- 
ments thaj/ certain lines or areas of fundamental research have a 
high level of potential relevance to education — and should be 
funded (and orchestrated) as such ; relevant to education. 

i 

Assuming for the moment agreement that funding should be provided 
(even increased) for fundamental research relevant to education, the 
question could be raised whether or not NIE should have a role in 
the funding process; and if so, what role. The NAS report has 
made a strong case that education has historically derived benefits 
from fundamental research in a viariety of disciplines — research 
funded for the moat part by agencies which are larger, more well- 
established than NIE and/or are more directly related to specific 
relevant disciplines (e.g.: NSF, NIH, NIMH). If this is the 
case, it would be reasonable to askVhy all federal funding for 
education relevant to fundamental research should not be channeled 
directly to such other federal agencies. In the absence of a 
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ratlonale £ov an NIE role, such would be a quite reasonable question. 
The NAS report lacks a clear and forceful statement of such a 
rationale — andvhereln lies a potential danger of the report. Th.e 
rationale for an NIE role is simply the strong need for a lead 
role of system building, orchfest ration and coalescing across disci- 
plines (as well as across the several major funding agencies) — 
and NIB has been mandated by the Congress to provide these l$»ad 
roles. ^ 

% 

.6. > The Use of Statistical Data 

When statistical data is used in any analysis, it is Important both 
to recognize the limitations of the data and to stay within these 
limitations when drawing conclusions 'or implications from the data. 
The NAS report does make initial disclaimers that (1^ the way a federal 
agency allocates its funding is a "far from perfect" measure of an 
agency's commitment; and (2) "the data needed to make comparisons 
lire sometimes unreliable or unavailable" (p. 50). We further recognize 
that the statistics used in the NAS report (Tables 1-6, pp. 51-59) are 
the most relevant available statistics. Nonetheless, in a number of 
instances, the report does use the available data in ways that appear 
to be inappropriate and misleading. This can be illustrated in Table 3 
(p. 55) and related discussion (p. 54) in the NAS report. 

As presented in the report. Table 3 includes dollar amounts that were 
allocated in FY 1975 to basic and applied research by a variety of 
federal agencies. Estimates are provided from two separate sources. 
Percentages are included only for sub-totals and totals — not for 
each separate agency. From Table 3, the report concludes that: "Both 
sets of data show that, overall, basic research receives about one- 
third of the total research support. This proportion is even smaller 
(22-29% of all research) in the agencies identified by the NRC Study 
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Project on Social RfieD as directly conterned with research in education. 
Basic research is smallest (15-20% ofi all research) In the Office of 
Education and the National Institute tof Education," 

V 

A more detailed analysts of Table 3 raises serious questions about 
these conclusions* and about the NAS conroittee's use of available 
statistical data. If the dollar amounts provided in Table 3 are 
translated into percentages (specifically: basic research as a 
percentage of total research allocations) , we obtain the results 
shown in Figure 1. When Figure 1 is used as a basis for a closer 
scrutiny of Table 3 than is provided in the NAS report, a number of 
implications- may be drawn that challenge the conclusions of the NAS report. 

f 

1) The NAS report concluded: "Basic research is smallest (15- 
20% of all research) in the Office of Education aiw^ the 
National Institute of Education" (p. 54). This statement 
is highly misleading in several respects. First, it com- 
pares OE and NIE with the average of thirteen other agencies » 
agencies which are significantly different in mission and 
whose individual percentages range from 0% to 100%. Second, 
w^pn NIE is considered separately, Eswlmate 1 places NIE 
quite high (e.g.: 47% for NIE cp. 37% for NIH; NIE, at 
47%, is higher than all of the other agencies except for 
NSF and Smithsonian). Third, as discussed elsewhere by 
us and by the NAS report, education has been a largely 
derivative field. Thus, NIE and OE do not represent the 
total enterprise for fundamental research relevant to 
education. ^ 



2) The NAS report concludes that the "average" percentage of 
total research funding allocated to basic research is 
33-35%. Granted that this is a mathematically true assertion. 
It is also a meaningless assertion. Individual agencies 
range from 0% to 100% allocation of total research funding 

ij; 
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Bstlnate !• Programs for Research 
in the Behavioral and Social Sciences 
(Naeional Science Foundation) 

Baale Rese ffV9^ Pere^ntaea of Total Research 



E|||inaCe 2. Research 
tM]«cta Relevant to Schooling 
and Other Formal Education* 
Selected Agencies Only 
(Interagency Coordinating 
Project) 



Agenciea Engaged in Sonie 
Education Research (Study * 
Project on Social R&D) 

DepartMot of HEW: 

Nati 'nal Institute 
of Ei'ucation 

Office of Education 

Subtotal ; Average 

Office of Aaa't 
Secretary of Education 

Health Division 

DepavLtneat of Agriculture 
Oepartaent of Interior 
Departoent of State 
Department of Coomerce 
Smithsonian 
Department of Labor 
Department of Defense 
Veterans Adminintration 
Subtotal ; Average 

Other Agencies Whose Research 
is Relevant to Education 

National Science Foundation 

Alcohol, Drug Abuse, and 
Mental Health Administration 

Health Services Administration 

Social and Rehabilitation 
Service 



^<«sic Research 
in Behavioral/ 
Socikl Sciencea 



A7X 
0% 
20X 

37% (NIH) 



24% 
11% 
0% 

under 2% 
100% 
4% 
20% 
5% 
(22%) 



72% 



44% 
-0- 

0% 



Basic Research 
la Education 



11% 
21% 
15% 

80% 

80% (NICUD, HINCDS) 

331 (OHD) 

73% 



0% 
26% 

(29%) 



7or. 

29% 
100% 

-0- 



Average 



33% 



35% 



FIGURE 1 

BASIC RESEARCH FUNDING AS A PERCENTAGE OF TOTAL 
RESEARCH FUNDING OF SELECTED FEDERAL AGENCIES 

^Adapted from Figure 3 In: National Academy of Sciences, "Fundamental Research and 
the Process of Education" (1977). 
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to basic research. It would be meaningful to compare NiB. 
with other specific agencies provided that the nature 
and relevance of the comparison are identified (e.g.: 
similarity of mission). However, this is not done in the 
NAS report. 

The NAS report attributes a high level of significance to 
the fact that both the "Estimate 1" and "Estimate 2" data, 
show basic research receiving about one-third of total 
research support as^ a^ composite average for alij. agencies 
combined . Given the above discussion and the fact that 
there is no data for several agencies in "Estimate 2", 
we question that this statistical fact has any real meaning. 

Finally, we would note the rather extreme differences 
between "Estimate 1" and "Estimate 2" in relation to 
several agencies (e.g.: NIE, OE, Health Division of 
HEW, Department of Agriculture). These differences 
highlight the limitations of available data and suggest 
the need for caution in drawing conclusions from the 
data. 

We may note briefly here another aspect of the discussion of 
statistical data in the NAS report". There are differences between 
federal agencies as to the aspects of the total R&D process for 
which they have concern or responsibililty. Since NIE has a respon- 
sibility for aU aspects of a total process (Including dissemi- 
nation and implementation/utilization), one must be very careful in 
comparing NIE funding data with funding data for agencies which 
have more limited areas of concern and responsibility. In a 
similar vein, in some fields we find the major funding for applied 
research, development, dissemination and even implementation 
utilization being provided by industry. In the education context, 
there is no such equivalent at least not on a comparable scale 
Thus, the demands on NIE's budget for (as an example) applied 



3) 



4) 
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research and development are likely to be stronger than the demands 
on the budget of (for example) NIH. These differences also make 
analogies difficult, imprecise and potentially misleading. While 
the data available to the MAS committee does not make comparative 
analysis an easy task, the NAS report does not seem to take these 
issues adequately into account in its discussion. 

7. Some Other Considerations 

The discussion thus far has served to illustrate a range of Issues 
relevant to the reports of the National Academy of Sciences and 
of the NCER Program Connlttee. \ There are additional issues which 
ne6d to be addressed^ Limits o\ time and resources prevent us 
elaborating on these for now* As illustrative of just a few of 
these issues we provide the following annotations of several points 
made in the NAS report* At this point we Will merely provide the 
appropriate quotation from the report together with brief comments* 
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A. Assumption: Educational Practice Has Improved 

We can also see that the quality of education as a human exper- 
ience has undergone marked Improvement over the years, not only 
between some distant historical point and now, but also within 
the lifetime of most adults. The curriculum of schools and 
colleges, for instance, has never been more varied in scope 
and variety^ High school students are learning now what was 
once thought to be college- level material, and elementary 
students are acquiring skills that used to be taught In high 
school. While some might argue that it has become too ambitious 
and that we should not be trying to teach so much to so many, 
there is no doubt that the varied fare that schools offer today 
is an advance over the three Rs of our grandparents* day.(PP» 17 



Does the discuscion of the NAS committee on pages IZ'-IS reflect a 
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rather "liberal bias" rather than a "disciplined inquiry" perspec- 
tive? There are those who doubt that today's varied educational 
fare is an Improvement (or at least has been at the expense of) the 
**three Rs*'. * Would not the *'back to basics'* movement be a direct (and 
widespread) challenge that there is "no doubt that the varied fare" 
. of today "is an advance over the three Rs"? Similarly, recent 
actions in La Crosse, Wisconsin (where the community recalled and 

defeated School Board members who had dismissed a "strong dlsclpli* 

\ 

narian" Superintendent) reflect what may be a more widespread and 
growing, txe^d back, towards strong internal school disciplinary 
practices^ Are the people who call for "basics" and "discipline" 
wrong? On the basis of what fundamental reseai;ch are they "wrong"? 
Even if one accepts the NAS committee's particular perspective of 
"improvements" in educational practice (improvements at least 
indirectly traceable to fundamental inquiry) , this perspective Is 
"hindsight". A critical Issue becomes how one can distinguish^ 
at any pj^esent moment, which lines of research and/ or related \ 
practice applications are "sound" and which are "fads" (and even 

r 

"bad" f ads) t How can the educational R&D co*mmunity inform or 
"warn" the practice based educational community about the limitations 
or weaknesses of implications which are prematurely drawn from the 
^tentative and/ or limited findings of a specific line of fundamental . 
research? These are pragmatic and value issues which are a part 
of the "reality" of the educational context but which are not 
addressed by the NAS report, 

B> Inappropriate Optimism 

These illustrative projects are not atypical of education 
R&D, nor are projects like them exclusive to education R&D. 
In our judgment, they repijesent an ill-advised tradeoff of 
scientific quality and future understanding for promises of 
immediate products and superficial benefits. To be fair, we 
must note that the promises have not always been made by 
administrators. Researchers themselves have sometimes approached 
their work with inappropriate optimism about the speed with 
which science might yield results that would inform practice. 
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Coriclusloa . The application of science and technology to 

improve education is of great impottance^ On the x-zhole, 

however, we believe that the federal government has adopted 

policies that encourage superficial and wasteful research that j 

has the appearance of relevance but lacks the substance of 

general principles* We recommend a significant redistribution 

of emphasis toward more fundamental research in education aud 

toward a more measured approach to education R6eD of all kinds^ 

The current resources for doing so are clearly sufficient, (p. 66) 

The NAS report is correct in challenging the validity o.f promising 
* or expecting "too much too soon" from particular fundamental 
rcs^^ch projects, and has also correctly noted that federal policies 
can have the effect, of encouraging such "inappropriate optimism'^ 
However, the report does not specif y^-^hich federal policies are . 
inappropriate --or how and why* Nor does the report analyze the 
political aspect of the educational R6d) context, identi>fy any other 
causeo for "inappropriate optimism", suggest ways of alleviating 
this problem. The report does seem to imply that a greater emphasis 
of -'fundamental research woulc^ "alleviate" this problem. We fail ^ 
to see the rationale or logic of this conclusion and indeed suggest 
that as a sole strategy, increased "emphasis" would not impact the problem, 

* ^ 
C. Limitations of Individual Research 



Federal agencies and the public are understandably concerned 

about the time required to solve problems through science or 

to get "answers" from research.. In pai^t, as we have observed, 

the outcomes of research have been misunderstood. But also, 

the time required to formulate and to carry out productive 

research is usually underestima'ted. We ^-mphasize .that this 

time cannot be reduced significantly by programming sequential 

activities, tightly supervising laboratories, dividing labor 

according to function, or "buying" clusters of research. The 

individual is at the hearf of fundamental research and he or 

she needs time to think, worry, and proceed with "gradualness. (p. 68) 

The NAS committee Vs primary concern here is to emphasize (1) the 
time and (2) the modus operandi for fundamental research. These 
are correct and needed emphases. We specifically agree that 
"programming sequential activities" and "tightly superyising labora- 
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tories'' are inappropriate. Neither do we challenge that the 
Importance of the role the ^'individual** researcher can have in the 
fundamental research process. However, ve do challenge the impli- 
cation that an "individual" approach is the only (or even the main) 
approach to fundamental research. Such an implication does not 
take cognizance of recent trends towards the use of teams 
in fundamental research in a number of areas (e.g.: in empirically 
based areas). . ^ 

We also question the report's conclusion against "buying"' clusters 
of research. Although the term "buying" has negative pejorative 
connotations, there an, reasons to consider funding "research 
clusters", whether such "clusters" be research areas or research 
centers. The earlier discussion in this paper about selection 
decisions has Implied that concentrating on a research area 
(tiirough a "cluster'.' of supported projects) may at times be 
prudent. Similarly, the need for system capacity building suggests 
a strategy of supporting "centers of excellence" i.e., insti- 
tutions where there is a "cluster" of fundamental research 
personnel. 

D> Unsolicited Proposals 

It is the Committee's opinion that the most productive nocl 
yet devised for managing research without destroying freedom 
of Inquiry is the research grant awarded after peer review of 
unsolicited proposals, (p. 69) 

There is no real "evidence" for this statement, which reads more 
like an "assertion" thpin an "opinion". Nor does the discussion here 
take into account how insoliclted and agency-suggested proposals 
should ba viewed in terms of issues discussed in this paper such 
as "lead" agency roles, agency/field relationships, selection 
decisions re. lines of research, etc. 
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IV. CONCLUSION ' y 

We sjtatttd at the beglnnlag o£ this paper that In two key areas, 
we strongly agree with the conclusions of the NAS and NCER 
Program Comnltteift reports: i.e., that fundamental research 
relevant to educiation (1) is important and (2) should be strengthened. 
At the same time, we have in this paper suggested some critical 
issues which have not been adequately addressed in either report. 
We have further suggested that consideration of these Issues Is 
necessary for the developioent of appropriate policir'S and 
atratagiea — and for the rationales and Justifications upon which 
such pollciea and atrategiea must be based. We have noted that 
consideration of a broader range of issues than those provided 
la the two reports leads, at times, to conclusions and recom- 
mendations dlffierent from those in the two reports; at other times, 
the result has been to support their recommendations and strengthen 
the rational la behind them. ^ 

It, has been our intent in this paper to be illustrative rather 
than comprehensive. Certainly, there are other issues which need 
to be considered and other policies and strategies which can be 
developed. We do believe that the' issues raised in this paper are 
critical issues and that these issues do demonstrate the type and 
range of issues — and related policies/strategies — which should 
be considered by NIE and NCER. 

As one looks across the range of issues discussed in this paper, 
three major requirements emerge that ever-arch the specific points 
of concern: 

1) that NIE' 9 policies and strategies' regarding education- 
relevant fundamental research must be based on a clear 
understanding of NIE's mission as a lead agency in the ^ 
educational R&D context; 
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2) that organizational and operational issues related to NIE's 
mission must be considered: 

/ 

/' 

3) that a variety of policiea and strategies (including but not 
limited to funding) foV strengthening education-re *«mt 
fundamental research need to be considered. 

1> The Mission of NIE as a Lead Agency in thet Educational R&D Context 

t 

The discussion in this paper points to a critical need for a clearer 
understanding of NIE's mission which can bei used to guide policy 
and strategy decision^ regarding all of NIE^s responsibilities — 
Including fiindamental research* The "starting point** and general 
framework of NIE's mission is, of course, provided by NIE*s author- 
izing legislation. At the same time, it is* Important that an agency 
such as NIE also take a proactive, self-defining stance in defining 
its mission. This at least involves: 

- determining the meaning and Intent which underlie the language 
of NIE* 6 authorizing legislation; 

- developing a vision of what NIE*s mission should be in light 
of a close examination of the nature of and needs in the 
education context; 

- ''spelling out*' NIE*s mission even more precisely in terms of 
directions; critical needs; relationship with NIE*s con- 
stituencies > other funding agencies, and the various education- 
relevant disciplines; operational implications; etc* 

Thus, while NIE obviously must start with it*s authorizing legis- 
lation as a general framework for detining its mission, NIE must also 
make a self-determination whether the scope of its mission should 
be more comprehensive or less comprehensive; what should fi'e the 
major foci or emphases at any given point in time; what *'mission" 

o 4 
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±8 appruprlate given the character of educati m (as a derivative 
fleldt as partially cra£t-like» etc*) and in the light of the state 
and stages of development of educational R&D in its various aspects; 
what la realistic within the constraints ot funding; 
and in this case what are the specific mission implications for 
fundamental research in education* 

Wa recognize, of course, that defining one's mission is not a _ 
^^prectse science", perhaps especially in the political context of 
a federal agency and in the va].ue-*laden context of education. 
Thus, any understanding or statement of NlE*s mission could (and 
likely will) be debated and would be subject to (Aiange over time. 
Further, we al^^o recognize that dt times a certain imount of 
"vagueness" about one's mission may be ''the better part of wisdom".* 

Nonetheless, we believe that NXE should give Immediate and serious 
attention to developing a clear Internal understanding and consensus 
of Its mission — as well as to developing such an understanding 
with ocher education-relevi^t funding agencies and with NIE's 
"constituents". Otherwise, NIE will continue to be subjected 
(internally and externally) t'^s^ non-resolvable and del^litatlng 
debates about policies and str^egles, to a lack of consistency 
and balance in its policies and strategies, and to periodic (and 
time/energy consuming) "rehashing of the same old issues ". 

» 

As part of an understanding of its mission, NIE must also give 
. early and serious consideration to what it means to be a lead 
agency. Certainly, this role is implied (1) in the very establlsh- 
ment of NIE as a* separate agency; and (2) in the establishment of 
a Federal Council on Educational Research and Development — composed 
of "represeptatlves of Federal agencies engaged in research and 
development relating to education" and chaired by the Director of 
NIE. 
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Further, consideration must be given to what it means to be a lead 
agency in the education context . As we have suggested, this will 
require consideration of such needs as system-building in a relatively 
immature, diffuse and highly "derivative" field; of orchestration 
roles in relation to other agencies, to the varied relevant disci- 
plines, and "participants" in educattonal R&D and educational 
practice; of the "practice-based" nature of education. Consideration 
should also be given to such issues as the extent to which there 
needs to be a "coxe" of researchers whose primary commitment is 
to the field of education. Finally, consideration should be given 
to the intersection and interaction of NIE's lead roles with the 
lead roles of other agencies concerned with educational R&D and with 
education-relevant fundamental research in particular. 

Considerations of mission, lead agency roles and the educational 
context do lead to specific policy/strategy implications and even 
requirements. While it is not our role in this analysis to determine 
what these will be, we have in our earlier analysis (Radnor, Spivak 
and Hofler 1976) provided illustrative scenarios of how these con- 
siderations could lead to specific planning and funding policies 
and strategies. 

In light of the above discussion, we recommend that NIE give 
immediate and serious attention to: 

1) defining its overall mission as a lead agency in educational 
R&D; 

2) determining the kind of leadership which is required and/or 
feasible for NIE' to fulfill its mission as a lead agency in 
the educational R&D; 

3) defining NIE's mission regarding education-relevant funda- 
mental research both In relation to education-relevant funda- 
mental research per se and to NIE's overall mission. 
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Xt Is our position that the deeper Issues Involved In determining 
NIE*s posture and programs with respect to fundamental research 
cannot be properly dealt with In the absence of these determinations. 

2. /organizational and Operational Issues 

If NI£ Is to fulfill Its mission regarding fundamental research 
relevant to education, early and serious attention must be given 
by NIE to a wide range of organizational Issuei^ concerning how NIK 
should be structured* how It should operate, how It should be staffed. 

For e3caiq>le: 

I . .. 

How Is NI£ to relate to/work with other funding ag^encles? 

How can NIE best keep In close touch with and work with the 
"field"? 

How can NIE obtain the information it will need to make selection 
decisions of what program areas to fund? 

How should NIE manage and supervise the projects it funds? 

Does NIE now have a program management capability which is- adequate 
and appropriate for its fundamental research mission? 

The point to be emphasized here, is that simply increasing funding 
for fundamental research without dealing with such organizational 
issues is not likely to be very productive. For example, if NIE 
does not have personnel who understand a particular research area 
and are "in touch" with the relevant research personnel, serious 
question must be raised as to what significant (or valid) role NIE 
could have in that particular research area. 

*\ 

Among th^ organizational issues v.hich NIE should consider would be 
at least the following; 
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1> having competent fundamental researchers on NIE's staff — 
and having them actively involved in research; 

2) determining appropriate roles for the field; appropriate 
ways of relating to the field; and appropriate methods 
for involving the field (e.g.: appropriate uses of panels, 
conferences, workshops, exchanges, etc.) in selecting, 
funding and working with those sub-specialties selected 

as targets for NIE support and attention* 

3) determining appropriate ways of working with other funding 
agencies in order to orchestrate. fundamental research efforts 
across agencies and disciplines; 

A) / determining appropriate uses of well-developed state-of-the- 
I art reviews, knowledge syntheses, etc.; 

5) monitoring the educational fundamental research context* 

/ ' \ 

' • " ■. ' ' 

Thi^s, we recommend that ixt planning policieJ, strategies and 

programs -for strjengthaning^education-relavant fundamental research, 

NIE give early and serious consideration to identifying and developing 

appropriate responses to. organizational issues such es (but not 

limited to) those noted above. 

3. Multiple Perspec t ives and Purposes of Funding 

Any examination of ^fundnmental research in the education context 
is llkeiy to reveal that education-relevant fundamental research 
shottld be strengthened and that NIE funding for fundamental research^ 
can be sotiewhat •♦.ncreasei? without "damaging" the balance of NIE 
funding for other areas of educational R&D. However, the funding 
issue needs to be examined from a variety of perspectives and in 
terms of raultiple-pu.:i: .se strategies. To illustrate: 
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1) ^ In determining levels of funding for fundamental research » 

\jlIE must first give consideration to the needs of and NIE's 
responsibilities in other areas of educational R&D and 
prai^tlce. Consideration must then also be given to the 
field's capability to use aiiid NIEU capability to "manage" 
particular levels of funding for fundamental research. These 

• considerations would seem to indicate that applying 20-30% 
of WIE's total budget to ftmdanental research would be un- 
necessary, inappropriate and potentially dysfunctional* As 

.we have suggested, a more realistic, strategy would be an 
Incremental, measured and monitored annual increase over 
NIE's current level of funding for fundamental research. 

2) Policies and stfrategles for strengthening fundamental 
research relevant to education should give consideration 
to such purposes as system building and orchestration 
across disciplines and funding agencies* 

3) Policies and strategies for strengthening fundamental 
research relevant to education should Involve careful 
selection of research areas for funding. In selecting such 
research areas » consideration should be given to whether 

or not they are: 

- areas which provide opportunities for orchestration 
and synergy across projects; 

areas where NIE can have a significant impact in 
terms of orchestrating the efforts of several funding 
agencies^ system buildings etc*; 

- areas whii-a informed judgment indicates potential 
significance for educe tlon; 



*By topics and by sub-specialties; It is our belief that disciplines 
and general fields represent a much too broad research spectrum. 
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- areas where "exceli^nce" exists or can be built; / 
^ areas where "seeding" is needed and cpuld be ji reductive « 

4) Recognition must be given to theNCact that the "genesis" of 
significant fundamental research tjiW come ftom applied 
research and ~.ln a "craft-like" fi^ld such as^^^catlon ~ 
from practice* v 

5) It may well be that the rationale for an Njlmmedlate Increase 
In funding levels is most validly based onXthe t^^ed for NIE 
to "signal" to the field NIE*s commitment . to^ fuxjdamental 
research and to -stability In NIE -s ability ^teid and manage ~ 

fundamental research. / 

/ 

/ 

It is imperative » then» that NIE view an increase In^ the level of 
funding for fundamental research as only one aspect /of a broad-*based 
funding policy. 

4. Sunnnary 

In conclusion, while we have said that we support an increase in 
NIE funding to fundamental research, ±% is our strongly held position 
that in the absence of specific resolution of and attention to 
required policies, resources, structures, and programs, such an 
expansion should be measured and incremental and not of the radical 
nature recommended by the NCER Committee (20% to 30% of budget) . We also 
oppose the use of such formulae for fundamental research funding policy. 

Among the areas requiring such specific attention are the identifi- 
cation of fundamental research areas in terms of current capabilities 
of the field in various areas ; potentials for synergy ; potential 
for developing "centers of excellence" ; potential for relatively 
"near term" significant results, etc.; organizational issues, 
including NIE's current program management Cdpabilities; defining 
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NIE's mission and the nature of what It means to be a lead B&D agency 
In the education context; the relative need for funding across the 
various educational R&D functions; whether or not there needs to 
be a "core" of researchers whose primary commitment Is to the field 
of education. 
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APPENDIX A 



Ve have Included as an appendix an editorial from the August 26, 1977 
issue of Science which discusses some of the same issues that we 
have reviewed in this paper. 
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26 August 1977. Volume 197, Number 4306 




How Fares Basic Science? 



When the elders gather to assess the spirit and substance of science, ap- 
prehension isjnvariably expressed about the state of basic research. Thus, 
it was only nomial for the recent summer meeting of the Committee of Sci* ! 
entific Society Presidents to vote that ''basic science is In trouble/' 

This is a useful, and suitably gloomy, battle cry to raise on the eve of the 
annual jousting exercise of budget-making. Unfortunately , the quality of the 
supporting evidence leaves nfiuch to be desired, fs basic science in grave ^ 
trouble, in significant trouble, or in some trouble? Is it equally in trouble in : 
government and in indvstry— or is there a great difference: a tale of two , 
cities? Whtt-trrthr properties of the trouble— financial, political, institu^^j 
tional, attitudinal, or managerial? How much of the trouble is self*inflicted? ; 
It would be helpftU to have answers. Lamentation is .^ot enough, and basic | 
science is not homogeneous^ . . . . - . . . 

As has been said before on this page, , basic science long ago drifted 
amiably into the arms of govemntent at the price^of those checks and bal*^ ! 
ances* which go with plunlistia support. It should not corneas a surprise to j 
leam:thtt the marginal industrial and fioundttsoo dollar has been driven out | 
of the picture, U has also been rot^ I 
very good fiiehd of basic s^ nee, everything cotisFd 
erratic and uncertain eavirbr At for long«term research because it has not | 
yet come round to treating basic science^as i/tveirmeni, in contrast to year- I 



ence. Instead of measuring ''trouble'* sbrictly in terms of rising, falling, or j 
steady-state budgets, we need to ask different questions and apply different ' 
tests. It may very well be that built-in inefficiencies and disuactions are . 
sapping the vitality of the research process and that the dollars allocated to : 
basic scienceno longer tell us much about the true levels of research effort. | 

WhicK fxhkt operate to devalue the basic science dollar? The indirect . 
cost surcharge on research grants is a familiar kind of burden, but not the 
only one. Count|ess man-months are subtracted from research effort in or- 
der to satisfy the routines of renewal application, accounting, reporting, and ; 
compliance with the rising tide of governmental and institutional regula- 
tions. Obsolescence-of instruments and equipment, together with queuing 
ddays» works to drop the blood- count of research. The torrent of what . 
passes for scientific and technical information presents obstacles through 
which investigators must blast dr tunnel their way. This is the enervating 
dimension of basic science as it must be practiced now, and little of it meets^ 
the unwary eye. Would it overreach by too much to say that, compered with 
the environment of a decade ago, the research dollar has been devalued^ in ; 
terms of productivity* by one^third? 

With zero-based budgeting coming on strong, and a balanced federal 
budget being scripted for 1981, prospects for growth of support for basic * 
science are problematic. The normal revenue growth under present tax laws 
will be claimed by defense, welfare reform, energy development, and per- 
haps a national health insurance program. Three-quarters of the govern- , 
ment*s budget is already relatively uncontrollable, leaving a very narrow 
area for discretionary e)cpenditure, and it is in this cramped and bitteriy 
competitive comer that basic science is to be found. Given all this, there 
are strong incentives for isolating those influences which undercut the pro* 
ductivity of basic research at existing support levels. 

Basic science has known lean times. But a combination of austerity, in- 
dustrial disinclination, continuing inflation, and falling productivity could 
prove to be too much. If the science adviser, the National Science Founda- 
tion, and the scientiflc community would take a close look at the issue of 
productivity, a brighter light might be shed on the sources of ''trouble** in 
basic science.— William D. Carey ^ 
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APPENDIX B 



A second appendix we have Included here Is a report to the 
National Institute of Education, (1977) entitled "Fundamental 
Kesearch and the Process o£ Educatlon'\ 
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The Committee on fundamental Research Relevant to Education, constituted 
in June'^lSye by the National Research Council in cooperation with the 
National Academy of Education, was formed in response to a request from 
the National Institute of Educatijin <NIE) . J3ne of the legislative chargai 
to the Instigate Is that it seek to Improve education in the United States 
by strenptaening its seientific foundations. In light of that charge. Dr. 
Harold llodgkinson. Director of the Institute, asked the Committee to rec-^ 
oinmend how that strengthening might be accomplished, by identifying promising 
lines of fundamental research, assessing th^ adequacy of federal support, 
and recommending changes in policy, if any, needing consideration by the 
National Council on Educational Research or other appropriate bodies. 

This Committee did not conduct a scientific research project. We 
were asked, because of our experience and expertise as scientists and 
educators, to express some Judgments about research and federal policy. 
We did not feel constrained— and were not asked—to suspend our initial 
belief in the value of fundamental inquiry for education. As persons 
who have committed our careers to fundamental research as well as to 
applied research and education, this belief was, and remains, strong. 
Nor did we feel it necessary to collect large amounts of new empirical 
data. There is much that we have learned over the years and much that 
others have learned. There exist numerous sources of information already 
available from reports, papers, and books on the atopic of research and 
■ education. Our task, as wj^cnceived it, was to review with each other 
our knowledge and perspectives and tp learn from documents and colleagues 
outside the Committee. Ou/ search for information and our discussions, 
while lengthy and to the point, were not so much exhaustive investigations 
as they were a form of shared reassessment of our judgments. 

The Committee did try to solicit comments as widely as possible and 
to familiarize itself with all the pertinent literature. For example, we 
systematically reviewed previous reports and evaluations of change in edu- 
catlon*and the effects of research (see Bibliography), undertook a limited 
citation study of the flow of research information (see Appendix A), and 
collected information on the performers of basic research by examining 
current journals, books, and nationally distributed magazines and the re- 
cipients of research awards. Some of us examined the research citti? in 
books that had had national impact on education. 
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The focus of much of our study w'^ps on research designed to understand 
the processes of individual learning ^6nd human .developtn.ent, the organiza- 
tion of social institutions,, and interpersonal interaction. We^reviewed ' 
only briefly applied research and work that translates basic reiearch into 
educational materials. Our major concern wit^ regard to applied rescjarch 
and development was to evaluate its scientific foundation. 

We did not evaluate fundamental research on the subject matter of 
education, such as mathematics and physics. This decision was largely 
due to our limited time and ex^ortise. During a one-day meeting, we did 
consult with a special mathema leal and physical sciences panel of th^ 
Committee, whose members were George Pimentel (University of California 
at Berkeley), Henry Pollak (Bell Laboratories), Frederick Reif (University 
of California at Berkeley), Frank Westheimer (Harvard University), Hassler 
Whitney (Institute for Advanced Study), and Bernard Witkop (National In- 
stitutes of Health). 

Our evaluation of research policy was focused mainly cn the National 
Institute of Education, because of its mandate to improve the scientific 
foundation of education. Our review of the Institute and its programs 
was as comprehensive as we could make it. We examined at length and in 
detail the current spectifxim of researpK^ support now maintained by the JJIE 
and familiarized ourselves with its-^torking structure. We interviewed 
program officials, examined budget documents and actual spending in detail, 
reviewed projects proposed to and supported by the NIE, and investigated 
provisions for maintaining scientific feasibility and quality. In addition, 
we reviewed the funding and management of educationally relevant research 
by the various governmental agencies that now offer such pupport. Among 
these are the Office of Education and the National Science Foundation. 
(During. our deliberations the latter began a new program for research in 
science education.) 

Our task in formulating and writing this report was made lighter then 
it might hi^e been because many individual^ helped. First, Philip Jackson, 
one of our own Committee members, deserves thanks. In an essay he wrote 
for the Committee, he captured our conception of the process by which re- 
search Is diffused. A revised version of this essay constitutes Chapter 2 
of this report. 

The Director and Associate Directors of the National Institute of 
Education and several other past and presevit program officials provided 
much of the assistance we needed. They were both cooperative and sympa- 
thetic with our requests for information, quick answers to questions, and 
discussion. We especially wish to thank Dr. John Mays, who, as the Science 
Advisor of the NIE, was responsible for transmitting to us the largest por- 
tion of the information and background materials needed. He was invaluable 
as a source of historical background and citations. 

Many others contributed at varioua stages of our uork. For example, 
our initial search for information on both rrsearch and policy was aided 
by the directors of a large number of associations, who announced to edu- 
cators and researchers our request for suggestions. In response to this 
request, we received over one hundred letters, visits, or calls. In addi- 
tion, Christopher T. Cross, a knowledgeable staff person from the Congress, 
spoke to our Committee about congressional views. The aforementioned 
panel of mathematical and physical scientists commented on the first draft 
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of this report and met with us for a day. Dr. David A. Goslin, Executive 
Director of the Assembly of Behavioral and Social Sciences, and Dr. James 
G. itiarch, of the NIE Council, also met with us and provided valuable sugr- 
gestions. There were a large number of unwitting helpers, too, among 
whom we count the originators ''of the reports listed in the Bibliography 
and the authors of books on educational problems and organizations. 

The revisions of this report have received careful scrutiny by re- 
viewers within the Academy. These reviews, many of which were written 
in great detail, were extremely helpful; we owe our thanks to these anonym 
Qous reviewers. We are grateful, too, to Eugenia Grohman, Editor and 
Executive Associate of the i^osembly,. who helped us structure the report, 
^nd to, Christine L, McShane, Assistant Editor, who critically edited the 
report and supervised its production. 

Finally, we wish to thank Benita A. Anderson, the Administrative 
Secretary of our Committee, who not only typed, the many versions of the 
report but was pur primary research assistant, collecting information 
from Journals and reports and collating by hand the large number of cita- 
tions used in the study reported in Appendix A. 

In conclusion, we joined the Committee with strongly ^jeld views atiout 
the nature of fundamental research and preferences regarding the lines of 
inquiry most worth pursuing. Another committee might have had somewhat 
different opinions. We tried to avoid narrowness on our Committee by 
reading and consulting widely, but we did not attempt a systematic survey 
of scientists and educators, nor did we try to achieve consensus on all 
aspects of the report. We do not presume to represent all our colleagues 
in this report nor to suggest that every part of this report represents the 
opinion of every Committee member. We did, in the end, find that the areas 
of agreement among Committee members were much greater than the areas of 
disagreement. As a whole, this report presents our collective judgment on 
those aspects of fundamental research relevant to education that we have 
been able to examine in the&te few months, and our recommendations regard- 
Ing research policy represent some hard thinking and reflection on the 
part of each of us. 

Sheldon H* I^hite 

Chairman^ Committee on Fundamental 
Research Relevant to Education 
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CHAPTEIjL 1 
INTRODUCTION 



The modern conduct of education — through schools, colleges, training pro- 
gramst television, publishing companies — touches every one of us in more 
ways, for more hours of the day, and probably with greater effect, than 
ever before. Possibly that is why education is so much in the news and 
is much of what seems tGt^ be behind the news. To many people, the quality 
and quantity of educatioti\seema connected with their own and their chil-- 
dren*8 chances for success ^important social problems such as unemploy- 
ment and crime, and the nation's stature. Our social, political, and 
economic ills and expectations transcend education, but the perceived 
Importance of "Rettinj? an education" to alleviate problems and achieve 
dreams is significant. Nearly all of the government's major social pro- 
grams have an education componei^« Th.erefore, when scientists and schol- 
ars turned their research toward the Understanding of education, people 
hoped for practical improvements in Instfuction as well as ♦"be .llevia- 
tion of apparently related societal problems* The questions sometimes 
raised by 'these hopes, and nourished by the e>^dence'of moon landings, 
antibiotics, and atomic energy, are how soon arid with what effect does 
research on fundamental processes bear practical^ fruit? These questions 
find pointed expression irt the phrase, "That's vAry interesting. Professor, 
but what's its relevance?" 

This report on fundamental research addresses the l33ue of relevance 
by aiming at three questions that we, a committee of scientists and edu- 
cators , believe are useful for a serious discussion of national research 
policy for education: "What do you mean by relevance?" "\NTiat kinds of 
fundamental research have potential relevance?" "How can federal policy 
strengthen fundamental research relevant to education?" 

Our answers to those questions take the following form: firsts what 
makes fundamental research relevant is the improved knowledge it gener- 
ates, which in turn is a condition for more useful views of how education 
takes place, new visions of what is educationally possible, stronger com- 
mitments by those who educate, and improvements in instruction and educa- 
tional institutions. Second, the kinds of fundamental research that have 
potential lelevance derive from a broad range, of inquiry focuRed on basic 
questions concerning how people mature, learn, and interact and how social 
ins*^itutions affect them. Third, federal policy for fundamental research 
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relevant to education should be designed or redesigned to improve the 
quality of work of those who conduct research and their working environ- 
ment, to enlarge the scope of fundamental inquiry, and to provide adequate 
resources for its developmer4t . 

These conclusions derive from our views of research and how it is 
administered. We believe that f .ndamental research relevant to education 
is basically a development of ideas for explaining how and why education 
occurs across places, time, ai.d groups of people. The quality of this 
development is reflected in the validity of the new concepts and under- 
standing that gradually diffuse to educators and the public, where it 
stands its ultimate test: the degree to which educators, students, and 
citizens find t^ie new ideas more useful, more sensible, than the old ways 
of thinking. In turn, the quality of fundamental research depends heavily 
on the standards of those engaged in it and on their resources for system-^ 
atic observation and careful analysis, bujlding upon the work of others, 
responding to emerging possibilities, and examining the many realms in' 
which basic Questional processes occur. These resources depend on two 
factors insufficiently represented in the practice of federal policy 
today— conmitments to financial support and flexible management that en- 
courage selfi-directed fundamental inquiry. 

Definitions ! 

Defining the subject matter of this report proved to be difficult. The ! 
colloquial definition of education is imprecise— to some it means schooling, ' 
and to some it means more than that. Our discussions of fundamental re- 
search relevant to education hinged on our agreeing on a definition of 
education itself. 

In Western society, the classical definition of education is intel- 
lectual development, or learning. The Latin origins of the noun, educa- 
tion, convey the notion of a leading out of ignorance, Plato recommended 
geometry as a course of serious study, not for the practical advantages 
it might afford in battle or eyeryday life, but rather because "geometry 
will draw the soul towards the truth and create the spirit of philosophy 
and raise up that which is unhappily allowed to fall down ..." (The 
Republic^ Book 7). Echoing this attitude over 2,000 years later, John 
Henry Cardinal Newman argued that the advantages of advanced education 
are "in one word, the culture of the intellect" {The Idea of a Vniveraity, \ 
1852) . ' ! 

Whilfe this definition of education is simple, it is not entirely I 
consistent with that of many Americans. In the minds of many, education ' 
means schools, colleges, and other institutions that, in turn, are called 
on to provide many functions other than intellectual development. Thus, 

education" can be viewed as a means for socializing children, ^ provid- 
ing day -care, vocational training, conferring status or credentials, 

^^Article 26 of the United Nations Declaration of Human Rights says. 

Education shall be directed to the . . . strengthening of respect for 
human rights and fundamental freedoms. .It shall promote understanding, 
tolerance and friendship among all nations, racial or religious groupsi 
and shall further the . . . maintenance of peace" (Dec. 10, 1948). 

j 
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aud stimulating national development. When, in a recent survey, parents 
were asked to rank the relative importance of various attributes of tneir 
children, intellectual curiosity lanked tenth (after characteristics like 
honesty and good manners) and success in school was tvelfth* Letters to 
the Committee from researchers and educators also showed a range of ex- 
pectations for educational institutions. In these letters, 40 percent 
emphasized that learning should be their primary function, but 60 percent 
mentioned training for occupations, good citizenship, mental health, or 
national development. These views suggest that if intellectual develop- 
ment were the only outcome of schooling, parents would not expect their 
children to spend at least twelve years In school, and the public would 
not spend an average of over $15,000 pe^ child on schooling. More to the 
point, it suggests that our discussion would be too narrow were we to 
focus on Intellectual "development alone. 

We have decided to use, for the purposes of this report, a two- 
dimensional definition of education. On one hand, education is personal 
and Intellectual development or learning, which may occur either inside 
or outside schools. On the other hand, education Is what educational 
institutions do, or are expected to do. It Is our belief that fundamental 
research is relevant to education to the extent that it leads to an under* 
standing of these domains. 

What is fundamental research? The Committee decided, arbitrarily, 
that there was no need to make a distinction between the traditional term 
•'pure science/* the popular "basic research," and "fundamental" research- 
Basic research need not be equated, as it once was by many, with labora- 
tory work or reseat wh conducted exclusively in academic departments. We 
believe It has com^ to mean disciplined research to discover general 
principles, but not necessarily by a particular academic or methodologi- 
cal route. Thus, for ercample, some of the work of psychologists on 
learning from Sesame Street, conducted on the site where the program was 
developed, is basic research, truly fundamental to uader standing how 
children learn. 

Fundamental research lu education is disciplinea inquiry whose pur- 
pose is to understand why and how education takes place. These processes 
are the subject matter of the behavioral and social sciences, such as eco- 
nomics, sociology, political science, psychology, and anthropology, and 
some of the humanities, such as philosophy and history. Our ability to 
comprehend the basic activities of education, to recognize and articulate 
problems, and to suggest ways and means for solvir; them depends heavily 
on the knowledge developed by these sciences and humanities. 

Objectives 

As a guide to the Committee's work, the National Institute o^ Education 
(NIE) asked us to consider three questions: 

1. What are the principal lines of research being pursued at the 
present ^-^me that are significantly strengthening the scientific founda- 
tions cf <;ducai:ion, and what are some of their possible contributions to 
American education? Are there some lines of research that appear particu- 
larly promising and deserving of higher priority than they are now given? 

2, Are current modes of conduct and support of fundamental research 
relevant to education adequate to ensure its quality and ultimate useful- 
ness to education? If not, how might they be improved? 
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3. In light of answers to the above, what possible additions to 
or chariges In policy relevant to fundamental research, if any, are rec- 
comnended for consideration by the National Council on Educational Research 
or other appropriate bodies? 

We began our examination of these questions by defining, operation- 
ally, the specific issues we would address and the array of work we would 
evaluate. Three major considerations affected our decisions. First, we 
were asked to prepare a draft of the report in nine months; second, we 
did not wish to repeat what other groups had done recently; and third, we . 
wanted to allocate our limited time to those issues we felt were most im- 
portant and about which we were most knowledgeable. 

The first question asked of the Committee by the Director of the NXE 
received considerable discussion and study. Implicit in the task re- 
quested — that of ideutifying lines of fundamental researdh having potential 
significance for education and describing research deserving of higher 
priority — was the more basic task of articulating how fundamental research 
makes a contribution to education. In short: How does education Improve? 
How does one define a "contribution" from research? Our reading of govern- 
ment documents and reports on education* the testimony of government offi- 
cials before Congress, and our discussions with congressional staff and 
program officials greatly increased our concern with these questions. We 
finally concluded that the usual evaluation of the impact of fundamental 
research knowledge on education is far too limited, and deserved our pri-.. 
mary attention. 

Education is a human service, a massive one. It does not change by 
leaps and bounds, and even when changes are Introduced by design, as in 
a new curriculum, one finds upon analysis that adaptation to the novelty 
takes place through a slow, complex, political process. Since clearly 
defined improvements in education are rare, it is also rare to find a 
direct and simple movement from fundamental research knowledge to educa- 
tional practice. And yet anyojae familiar with schools, school management, 
teacher training, and parent-school relationships knows of the movements 
that have taken place from disciplinary knowledge to public discussion, 
curricula, and teacher beliefs, which ultimately define practice in edu- 
cation. We found that many reports, program guidelines, and budget docu- 
ments reflect a far more limited perspective. Presumed In these written 
materials and the words of many government officials who spoke with us 
was the conviction that there must be identifiable change that clearly 
results from a well-defined, once-articulated set of ideas. 

We therefore undertook to examine a subtler and deeper vein of trans- 
mission from knowledge to education. We believe that educational change 
is slower, more subtle, and more complex than that usually envisioned, and 
that One of the most Important influences that fundamenta.! research has on 
education comes through diffusion rather than dissemination. Our concep- 
tion of th^is diffusion process is discussed in Chapter 2. 

The consensus we reached as a committee on the contribution of basic 
research to education had considerable impact on a subsequent decision to 
restate the first question directed to us as: How does fundamental re- 
search contribute to education? It was our judgment that ideas from basic 
research flow gradually, and in complex ways, to the educational community. 
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citizens, parents, and students. Theie \Ldeas affect not simply educational 
techniques but the way people think about education, the criticisms and en- 
thusiasms they have regarding it, and the aspirations they hold for them- 
selves and others. Some of this influence can be foreseen, roughly, in 
fundaaeritaX research as It progresses, but much of it cannot. W& therefore 
fe!|.t- it inappropriate to rank specific lines of research that might make a 
contribution to education. Instead, we attempted to delineate, by example, 
fundamental inquiry that has potential usefulness for education. We wished, 
through these examples, to illustrate the variety of methods that can be 
used to address topics and problems relevant to education — the process of 
building the scientific foundation of education — and to demonstrate the 
potential of contemporary basic research. .These examples are found in 
Chapter 3. 

We had also a more general task of evaluating the range of basic re- 
selarch and the health of work relevant to education. The. earlier work of 
individuals and groups aided in this task. One of the mo&t thoughtful 
volumes we read was Reeearoh for TomorroD's Schools^ edited by Cronbach 
and Suppes. We also read a large number of papers written by working 
groups of researchers who .had been sponsored by the NIE. Reports by dis- 
tinguished groups identified interesting and promising lines of reserach 
in neuropsychology, information processing, cognitive development, social 
development, linquistics, sociology, anthropology, and various kinds of 
learning difficulties. We discovered, in addition, a large number of 
literature reviews in the various disciplines that identified important 
problems on which excellent research was being conducted. Finally, we 
considered a series of other reports, some sponsored by the National Re- 
search Council, that evaluated and listed promising lines of research. 
(These sources are listed in the bibliography.) Once we were fully aware 
of all this previous work, we concluded that the question of what research, 
in particular, might be usefully supported had already been adequately 
answered, at least for the time being. Promising topics for fundamental 
research have been laid down, if not to the complete satisfaction of all 
the Committee members, then in abundance and with sufficient regard for 
quality and promise. Furthermore, we think that Identification of prom- 
ising research must, as a general practice, be based upon the implicit 
guidance that derives from the system of peer review. 

The second and third questions directed to the Committee by the 
Director of the NIE concerned federal policy, which we were to evaluate 
and about which we were 1:6 formulate recommendations. In I this effort we 
were guided by two assumptions based upon training and exi!)erience. First, 
research is only so "relevant" as its quality allows. If research is not 
of high quality, no amount of apparent pertinence to important: educational 
issues in its content , method, or site of study will make it relevant. 
Therefore, research policy must enable the research environment to promote 
quality in the work of researchers whose support derives from the federal 
government. Second, today's research cannot be conducted, on the whole, 
without financial support from the federal government. The problems are 
far too complex and numerous, the facilities required are too expensive, 
and the training needs are too sophisticated. Thus, we examined research 
policy with attention to thp adequacy of funding as well as to the quality 
it encouraged. 
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Our evaluation of research policy, described in Chapter 4, provoked 
much discussion and resulted in the recommendations ending the report. 
We hope these recommendations communicate our continuing belief that the 
federal government can and should support the growth of knowledge about 
education that we need to alleviate its problems, to build upon its 
strengths, and to shape it for the benefit of fuf ce generations. 
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CONCLUSlL S AND RECOMMENDATIONS OF THE COMMITTEE 



In this report, the Committee on Fundamental Research Relevant to Education' 
has set forth its views of the contribution that fundamental research can 
and has made to education. That is, fundamental research has had its 
major and most useful impact on education through the gradual, public dif- 
fusion of new ideas and concepts that have been assimilated into the ^ 
expectations, practices, and resources of education. These have influ- 
enced practitioners' views of reality, their vision of the achievable, 
their know-how, and their commitment to act (Chapter 2) . We have described 
briefly by example the kinds and variety of fundamental inquiry that we 
believe may make such a contribution in the future (Chapter 3) . Me have 
noted that federal policy in practice does not emphasize fundamental 
^research (Chapter A) . Our recomniendatlons are made with the hope that 
the federal government will reorient operating policy in education, toward 
fundamental research on how people learn and mature, their diverse sources 
and settings for learning, and the function and value of what they learn 
as well as toward improving the quality of all efforts to improve or 
alleviate problems in education. 

A Reemphasis on Fundamental Research 

1. Fedetal policy to build the scientific foundation of education 
through fundamental research is established in law, precedent, and concept. 
Nevertheless, basic research on the processes of education is today 
assigned very low priority in federal agencies charged with the manage- 
ment of educational research and development. In federal agencies, gen- 
erally, basic research receives about 11 or 12 percent of all funds for 
R&D; in education, basic research is allocated only 4 percent. In the 
two agencies primarily concerned with public education in this country— 
the Office of Education and the National Institute of Education--basa.c 
research receix s less than 2 percent of the research and development 
monies. We recommend an increase in the proportion of the federal in- 
vestment in education research and development designated for fundamental 
research (p. 66). 
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Improving SclenCific Quality of Research and Development 

2. Government agencies have swung toward premature attempts to pro- 
vide quick solutions to educational problems, many of which are not well 
understood. It is our conclusion that without the guid?ince of understand- 
ing, these practices regularly lead to projects that are of neither practi- 
cal nor scientific value. We reoormend a change in polioy touxucd more 
careful aeseeement of what ia knom and what must be teamed when solution- 
oriented programs are undertaken <p. 66). ^ 

3. Agencies concerned vith educational research are properly con- / 
cerned with setting research priorities and objectives. But too often 
the felt significance of an educational problem has been the overwhelming 
factor in allocatiug research effort, with insufficient regard for the 
scientific feasibility of the proposed research. We reoormend that more 
active inveatigators be included in the planning and program review of all 
basic and applied research efforts in education (pp. 67, 68). 



Better Management of Fundamental* Research 

4. Management practices that have proved appropriate for developing 
new curricula and moving technical advances into the educational system 
have not been particularly appropriate for strengthening basic scientific 
research. We recommend more extensive use of fieldfdnitiated and peer^ 
revised systems of research funding (pp. 69, 70). 

5. For some of their programSj the National Institute of Education 
ani other agencies use a sin£\e review panel designed to serve different 
objectives, such as to improve scientific understanding, to encourage iria- 
terials development, and to devise applications. This practice leads to 
overload, watered-down concentrations of competence, and a .tendency for 
the more applied s and immediate problems to preempt totally >the resources 
available. We reoormend that xoithin each major program (such as Bas%o 
Skills in the National Institute of Education or the Office for Handicapped 
in the Office of Education) s separate budgets and review panels be estab- 
lished for field-initiated research. Review panels should be staffed pv^- 
dominantly by currently active basic researchers, with appropriate 
representation of those more oriented to development and application 

(pp. 70, 71, 76). 



A More Active Role for the National Institute of Education 

6. The National Institute of Education has not made significant pro- 
gress toward fulfilling its mandate to strengthen the scientific and 
technological foundations of education. We recommend that the National 
Institute of Education take immediate steps to implement a policy of 
strong support for fundamental research relevant to education (p. 75). 

7. The National Institute of Education should offer leadership in 
fundamental and applied research relevant to educatxun. We recommend that 
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'the Institute redefine its role and implement policies to attrabt ami .nain- 
tairt ressarah of high quality in the field of education, to provide long- 
tem support for work on important problems of education that affect broad, 
sectors of sc^ety^ and to encourage pioneering applied and fundamental 
research (p. 75) ♦ 

t 

8. The National Institute of Education xiow limits itself almost 
exclusively to education in public schools. We recommend that its mijsion 
be broadened to include sponsorship of fundamental research on learning 
throughout life and in the many settings in which education , occurs (pp. 75, 

9. The staff of the National Institute of Education must be well 
informed about research. Ve recofwnend that the National Institute of 
Education adopt personnel polio-u^s that will facilitate the staff *s knowl- 
edge of research and of programs for research (p. 71, 72). 

National Sc:^^nce Foundation Participation 

10. - The Science Education Directorate of the National Science Foun- 
dation Is now planning its first deliberate program of support for research 
on science education. The National Science Foundation should establish a 
strong program of support for fundamental research related to science edu-^ 
cation (p. 75). ' 
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APPENDIX C 



A third appendix included here is report oy the 
National Council on Educational Research on Fundamental 
Research Relevant to Educat'l^.on . 
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NATIONAL COUNCIL ON EDUCATIONAL RESEARCH 

NCER Program CcTnmittee Report 
an4 Draft Policy Resolution . . 

on 

Fundamental Research Relevan t to Education 

The Nationai Institute of Education was' estjablished in 1972 in order to increase 
the effectiveness of educational research and development under the policy of 
' United States to provide for all persons opportunity to receive an education of 
high quaUty. Congress recognized the relationship between quality of education 
and research of a fundamental nature in the following statement (in the General 
Education Provisious Act of 1972) : 

"To achieve quality will require far more dependable knowledge about 
• the process of learning and education than now exists or can be expected 
•• from present research and experimentation in this field. While the 
direction of the education system remains primarily the responsibility 
of state and local governments, the Federal Government has a clear 
responsibility to provide leadership in the conduct and support of 
■ scientific inquiry into the educational process." 

The Institute was given full responsibility under the Act of 1972: "1.) To 
help solve or alleviate the problems of, and to promote the refom and renewal 
of American Education; 2.) to hexp advance the practice of education, as an art, 
sciencei\and profession; 30 to strengthen the scientific and technological 
f ov-nd .u. -ons of education; 4.) to build an effective educational research and 
development "system." Scientific inquiry into the educational process, or research 
of a fundamental nature, is essential in carrying out each of these responsibilities, 
and pursuit of the third depends alaiost entflrely on this kind of research. 
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The Council requested the Director to contract with tlae National Academy of 
Sciences, in cooperation with the National Academy of Education, for a study of 
fundaaiental research relevant to education. To paraphrase) the Academy report 
^ Fun'^^^^nt^al Research and the Process of Education fundamental research in edu- 

cation is disciplined inquiry whosi purpose is to understand why and how education / 

- 5, 1 

takeJ3 place*. These processes are the subject matter of the behavioral and social 
sciences, such as econoiaics# sociology, political science, psychology, and 

^ anthropology, eind some of the humanities, such as philosophy and history. The 

A' ■ 

relevance of fundamental resejarch to education derives froA the insights Xt provides 
into learning, wherever it occurs, and into schooling, tliat aspect of learning 
which is more formally organized. We see the Committee's formulation as quite 
« consistent with the definitions used by the Nationad. Science Foundation: 

> 

"In basic research the investigator is concerned primarily wiiih gaining 
a fuller knowledge or understanding of the subject under study." 

> 

••In applied research the investigator is primarily interested in a 
practical use of the knowledge or understcinding for the purpose of a 
recognized naed." 

The Academy has concluded tiiat "basic research on tlie processes of education is 
today assigned very .ow priority in Federal agencies charged with the management 
of educational research and development," and that as this applies to the 
National Institute of Education, the Institute "has not made significant progress 
towaurd fulfilling it? mandate to strengthen the scientific and technological 
foundations of education," The Report suggests that serious problems nay' flow 
from this sit'jation. 
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If the knowledge upon which applied research and development projects are based 
is inadequate, the likelihood of their producing valid or beneficial results is 
doMiftl. Although the Cormittee is aware of recent progress ^in NIE support- for 
fundamental research, it believes that even more must be done. If fundamental 
research is neglecte.l, the program of the Institute becomes less effective. 
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A major part of the Institute's, support should continue to be for projects that 

hava a direct benefit for educational practice and for research of an applied 

( 

rather than a fmdamenjal nature, including problem-solving, develoi' .rt, and 
dissemination. A considerable body of fundamental knowledge already exists upon 
which to base this kind of research. Neverthel;.ss, we are concerned that the 
knowledge base that already exists should also be extended in order to continue, 
to make application and development possible. 



Wei'.have consulted with a number of scientists and educators, both about the 
National Academy's Report and about their own views of the need for fundamental » 
research in education. As a result of these consxiltations , the Academy Report, 

and o'or own de.l5Jberations over ?. period of several months, we dre convinced that 

1 

the situation described is serious, that its implications for the future are even 
more serious, and that NXS support of fundamental research relevant to education 
should be increased substantially if ^ the Institute is to fulfill its potenticil for 
aiding major long term imp. /'uieat m education. The following policy is therefore 
proposed. 



\ 

!• Allocatiot) of Funds^ - 

A* At least 20% of the Institute's funds shall be used to support 
fundamental research relevant to education by FY 1979, and by 
FY 1985 this shall have^ increased to at least 30%» 

B. At least half of the minimum thus allocated (10% of the Institute's 
funds in FY 1979, increasing to 15% by FY 1985) shall be used for 
research grants to individual investigators or small groups cf 
investigators. 

II. Research Conanunitv Ijnvolvement; 

»■ 

A. The major pxarpose of research supported under thi? "X)lic> shall 
Le to extend €he knowledges gained through previous or cxirrent 
research or to e:^lore areas where knowledge is lacking and for 
which a need is evidejcit. Since those ,wHo are most 'familiar with 
/such areas of research *are investigators in the field, the Institute 



*In pXwlininary discussion by the Council some members, while supporting 
fully the intent of the Committee's recpiraxuendations , preferred the following 
wording of Section I: ' 

' . . ' 

A. A significantly greater portion of the Institute's budget shall be 
used fbr fundamental research in FY 1979 than in FY 1978 and each 
year thereafter until a level is reached tha^ is ' satisfactory to " 
both the Director and the Cc^icil. 



3. At least half of the funds thus allocated shall be used for research 
grants to individual investigators or Small groups of investigators. 




'shall, to the BiaxUnua extent possible/ rely upon their assistance 
in identifying research needs and research to be supported. This 
assistance should include, but not be limited to, submission of 
unsolicited proposals', participation in developing guidelines, and 
review of proposals by groups of highly capable investigators 
(with multi-disciplinary and cross-disciplinary membership) , 
including investigators close to educational practice. 

B. As part of the continuing external . review of NIE, the scientific 
quality and balance of fundamental resesurch programs shall be 

, evaluated regularly by outside investigators, 

C. The Director shall seek to work w'.th all major research associations 
and orgsuxizations whose members may be engaged in fundamental 
research related to education to accumulate knowledge about such 
research ana to c-ordinat the Institute's support program with such 
research. 

Coordinationj ^ 

The Director shall seek through the Federal Council on Educational Research 
and De'/elop-«ent and other means to coordinate fundamental research 

t 

t 

supported by the Federal agencies, to identify areas of fundamental 
research which should have greater support by NIE or another Federal 
agency, and to stimulate fundamental research in such cLceas. 
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IV, Staffs Respcpsibility For Fundamental Research: 

The Director shall insure that within each Program. Group there is a 
substantial nuciber and proportion of staff members who have competence 
and. recent (experience in fundamental research, and that these reseeurchers 
shall be inv^ved in positions of responsibility in all activities 
of the Program' Groups affecting furdamental research decisions. 

V. Reports t 

IniticLL plans for implementation of this policy shall be convvsyed tc 
the Council within 30 days of its adoption, including resources, 
allocations and management strategies. Similarly, by January 1, 1978, 
the Director shall present to the Council for its concurrence a general 
plan for implementation of this policy through 1985. 

The Director shall include in each annual report to the Council a 
review of the Institute's fundamental research work and efforts to 

improve it, and shall present to the Co"nciI as part of the budget 
planning process an analysis of option, for support of fundamental 
research under this policy. 
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